Karadi Path Magic English SLL Level 1 Programme MidlineAssessment: Samanvaya Foundation

Karadi Path Education Company, Pvt. Ltd.

Report generated on Tuesday, 29 April 2014

Contents

1 Executive Summary 2

1.1 Summary of Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Karadi Path Magic English SLL 4

2.1 Training and Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.2 Implementation Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3 Assessment Design 6

3.1 Sections and Weightage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3.2 Assessment Design and Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

4 Assessment Results 7

4.1 Aggregated Results, by Assessment Component . . . . . . . . . . . . . . . . . . . . . . . . 7

4.2 Aggregated Results, by School . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4.3 Aggregated Results, by Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4.4 Disaggregated Performance Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4.5 Histograms, by School . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

List of Figures

1 Overall Marks Distribution Across all Schools, Boxplot . . . . . . . . . . . . . . . . . . . . 3

2 Overall Marks Distribution Across all Schools, Histogram . . . . . . . . . . . . . . . . . . 3

3 Baseline and Midline Mean Performance Comparison, by Component . . . . . . . . . . . . 7

4 Baseline and Midline Performance Comparison, by School and Component . . . . . . . . . 9

5 Baseline and Midline Performance Comparison, by Class . . . . . . . . . . . . . . . . . . . 10

6 Histogram of Total Comprehension Score by School . . . . . . . . . . . . . . . . . . . . . . 13

7 Histogram of Total Speaking Score by School . . . . . . . . . . . . . . . . . . . . . . . . . 14

8 Histogram of Total Reading Score by School . . . . . . . . . . . . . . . . . . . . . . . . . . 15

9 Histogram of Grand Total Score by School . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1

1 Executive Summary

During the 2013–2014 academic year, Level 1 of the Karadi Path Magic English SLL programme wasimplemented in four schools supported by Samanvaya Foundation: Jakkasandra, KSRP, Satya NarayanaSchool, and Hombegowda.

To understand the impact of the implementation of the programme, a baseline assessment was done inJuly 2013 and a midline assessment was completed in March, 2014. The assessment was comparative innature—it was designed to assess the level of the students at two different points in time. The purpose ofthe assessment was not to directly assess each child, per se, but to attempt to assess the impact of theKaradi Path Magic English SLL programme on comprehension, speaking, and reading.

The assessment was designed such that:

1. It is an objective assessment.2. It is easy to implement and does not require many resources.3. It should not require too much time per child to complete (a typical assessment takes between 10

and 15 minutes to complete).4. It should be reflective of the language development expected to result from the programme’s

implementation while not covering information from the programme directly (thus eliminating thechance for a student to memorize a set of answers).

The assessment plan for the midline assessment was such that on each day, one school would be assessedin the morning session and one in the afternoon. The assessments were carried out by Ms AnithaaNagaraja and Mr Udhaya Kumar of Karadi Path Education Company, and Ms Mala and Ms Kalpana ofSamanvaya Foundation; Ms Kalpana was available only for the morning assessments. All of the peopleinvolved in conducting the assessments had completed this assessment before, either during the KaradiPath programme baseline assessment for Samanvaya Foundation or for a baseline assessment at anotherKaradi Path user school.

While the baseline assessment covered 103 students, the midline assessment covered 87 students. Thenumber of students common across both assessments was 72. Since this is a comparative assessment,only those students who were present at both the baseline and midline assessments were considered whenpreparing this report (see Table 1).

III IV V VI VII VIII Sum

Hombegowda 0 0 0 0 0 9 9

Jakkasandra 3 3 3 3 9 6 27

KSRP 2 4 4 5 4 0 19

Satya Narayana School 4 4 5 4 0 0 17

Sum 9 11 12 12 13 15 72

Table 1: Number of Students Participating in the Assessment, bySchool and Class

1.1 Summary of Findings

The grand total score distribution at the time of the midline assessment ranges from 13 to a maximumof 140 out of 140. A similar range was observed in the baseline assessment. The boxplot (Figure 1)and histogram (Figure 2) both show a positive overall shift towards higher scores and greater levels ofconsistency in performance among students.

However, whereas the range for the middle 50% of the students was between 18 and 53 in the baselineassessment, the inter-quartile range in the midline was between 63 and 124. As is evident in the boxplot

2

Figure 1: Overall Marks Distribution Across all Schools, Boxplot

in Figure 1, there was almost a 180-degree pivot around the 50 marks score. That is, approximately79% of the students scored marks greater than or equal to 50 in the midline. By comparison, during thebaseline assessment, approximately 74% of the students scored marks less than or equal to 50 overall.

The grand total mean score at the time of the baseline assessment was 27.25 compared to 63.72 at thetime of the midline assessment. This represents a change of 233 %. This change was achieved with 72sessions of 40 minutes each (48 hours) of implementation of the first level of the Karadi Path MagicEnglish SLL programme.

Figure 2: Overall Marks Distribution Across all Schools, Histogram

The greatest change by section (see Table 2 in Section 4) was observed in the speaking scores, where themean scores increased by 43 percentage points. Reading increased by almost 31 percentage points, whilecomprehension increased by just over 20 percentage points.

3

2 Karadi Path Magic English SLL

The Karadi Path Magic English SLL programme is a two-year immersive English language programmethat attempts to rapidly improve the English proficiency of children from predominantly non-Englishenvironments. It does so using a process that tries to replicate many of the elements of mother-tonguelanguage acquisition in a systematically condensed programme.

The programme is designed around four modules:

• Action Path: In this module, students learn sentence structures and vocabulary through a seriesof imperative statements. The sentences range from single word commands to complex compoundsentences which demonstrate proper use of many of the components of language that are difficult toobtain without extensive practice.

• Music Path: This module uses the familiarity and enjoyment that a child has when listening tomusic as a foundation to develop language skills. The songs have been carefully written to createmany opportunities for conversations and vocabulary development, all in a stress-free environmentfor the child and teacher.

• Reading Path: As it is expected that students from non-English environments would not havehad much exposure to reading, the Reading Path module starts with introducing students to thebasic phonetic sounds of the English alphabet, and progresses to building phonic and sight wordrecognition in the students. The module is designed to rapidly give students confidence in theirword recognition ability with the phonic words, but quickly accelerates into development of sightreading.

• Story Path: The Story Path module tries to replicate as much of the mother-tongue learningprocess as possible in the classroom. In the mother-tongue approach, we use our voice expression,facial expression, gestures, and objects to help a child develop language proficiency. Similarly, inStory Path, the key tools used are professionally narrated stories with contextually appropriatebackground music (voice expression); a range of activities including miming, conversation practice,and tiny theatre (facial expression and gestures); and vividly illustrated story books (objects).

The entire programme is designed such that:

• A child should not need an explanation of a grammar rule or a definition of a term. The context inwhich such things are presented should be sufficient for the child to be able to derive the meaningon their own.

• A child who goes through the programme would be equipped with both the skill and attitude ofcontinuously improving their language skills from whatever they are exposed to in their environment(television, newspapers, and so on).

• The programme should place low demands on the English language proficiency of the teacher. Thesession plans and trainings offered to the teachers specify what the teacher should do, when theyshould do it, and how it should be done. This also reduces the burden of a teacher having to preparesystematic lesson plans with a long-term language development perspective on their own.

• The programme should not place too much of a demand on expensive infrastructure. All that isrequired is a relatively quiet place and an audio system capable of playing audio CDs loudly enoughthat all children in the class can hear the audio clearly.

2.1 Training and Support

Karadi Path provided training and support to the teachers employed by Samanvaya Foundation whowere handling the Karadi Path Magic English SLL programme in the four schools. This included aninitial training as well as subsequent training of any new teacher who was appointed to handle the MagicEnglish sessions.

There was regular phone and email communication between members of the Samanvaya Foundation teamand the Karadi Path Education Company team to ensure the smooth progression of the programme’simplementation.

4

In addition to the training and electronic communication, on-site support was offered in the form ofclassroom observations, at which point detailed feedback was given to the teachers to guide them inimplementing the programme for the best benefit.

Because of the nature of the programme, this offered a good opportunity for both Karadi Path EducationCompany and Samanvaya Foundation to be involved in a programme implementation assessment (describedin Section 3). For this, special orientation was provided to those conducting the assessment.

2.2 Implementation Scenario

The programme was offered at four schools already being supported by Samanvaya Foundation. One ofthe areas that Samanvaya Foundation focused on was trying to improve the spoken English skills of thestudents. Although a few of the students at the schools they support study under an “English medium”section, even in those cases, the majority of the instruction took place in Kannada.

In selecting the Magic English SLL programme, Samanvaya Foundation was looking for a solution thatwould:

• Offer some means of assessing the impact of the intervention.• Be easy to implement by the teachers that work with the Samanvaya Foundation.

The teachers who handled the Magic English programme would work on a regular programme implemen-tation schedule. In one day, for instance, they might conduct the Magic English sessions class-by-class inthe morning at one school and proceed to another school and conduct the classes in the afternoon there.

The materials used by the teachers were provided as part of the “School Kit” provided by Karadi Path.This included a flip-chart and audio CD for use with Reading Path, a set of audio-books (book plusaudio CD) for use in Story Path, and a book of Karadi Rhymes with accompanying audio CD for use inMusic Path. The content for Action Path as well as session-by-session implementation instructions wereincluded in the Magic English SLL Teacher’s Manual.

A personal portable CD player (discman) along with a set of portable speakers was used for playback ofall the audio materials.

5

3 Assessment Design

3.1 Sections and Weightage

There were three main sections in the midline assessment, each with 2 or more sub-sections. These are:

1. Comprehension (20 out of 140 marks, approximately 14% of total). The listening comprehension ofstudents was assessed with two tests.

1. Action Commands: Students were given a set of 10 imperative statements to perform andtheir ability to understand was recorded according to their correctness in their actions.

2. Point and Answer: Students were shown two pictures and were asked questions about them.Student answered by pointing to the correct objects being asked about.

2. Speaking (80 out of 140 marks, approximately 57% of total). The speaking skills of the studentswere assessed by looking at their response to questions based on the two pictures mentioned above.Their response again was categorised into two sections.

1. One Word2. Phrase or Sentence

3. Reading (40 out of 140 marks, approximately 29% of total). The reading fluency of the studentswere assessed using four progressively graded reading tests.

1. Stage 1a: Single words, L12. Stage 1b: Single words, L23. Stage 2a: Three word sentences.4. Stage 2b: Five word sentences.5. Stage 3: Longer, more complex sentences.

For convenience, the scores for “Reading: Stage 1a” and “Reading: Stage 1b”, and similarly, for “Reading:Stage 2a” and “Reading: Stage 2b”, have been summed before taking the mean.

3.2 Assessment Design and Protocol

The Karadi Path programme assessment is objective by design. The assessor does not ask questions thatrequire subjective answers such as the students talking about themselves or their families. The completequestion and answer keys are provided to the assessor who is expected to only record the linguistic qualityof the responses.

The Karadi Path programme assessment tests language understanding and usage in contexts that areunlinked from the Karadi Path Magic English SLL material. While the test is based around the vocabularyand language usage that the students would have encountered during the programme, the linguisticconstructions of the answers are not found in the text for the students to memorise and repeat. Theassessment tests if students are able to generate English sentences independently in new contexts usingthe vocabulary and grammar that they have learned with Action, Music, Reading, and Story Path.

Based on the above and based on the limitations of resources available, some fundamental premises inthe design of the Karadi Path assessment protocol were as follows:

1. The assessment was simple and easy to administer and record results.2. It was realistic to the domain of students being assessed.3. The assessment showed pre-programme and post-programme levels of the students for English,

specific to understanding and speaking. Hence the same test was administered at the baseline andmidline to assess programme impact.

4. It was completely objective since multiple assessors would be administering the tests to the students,and a significant period of time would have elapsed between the baseline and endline assessments.

6

4 Assessment Results

4.1 Aggregated Results, by Assessment Component

Table 2 and Figure 3 look at each component of the assessment separately. That is to say, the weights foreach section are not applied in the calculation. The results are represented as the mean of the percentageof total possible for that particular component of the assessment.

Baseline Midline

Comprehension (Mean) 63 83

Comprehension (Median) 65 90

Speaking (Mean) 16 60

Speaking (Median) 12 62

Reading (Mean) 31 62

Reading (Median) 20 72

Grand Total (Mean) 27 64

Grand Total (Median) 22 65

Table 2: Mean and Median Performance Comparision by Componentand Assessment

Figure 3: Baseline and Midline Mean Performance Comparison, by Component

7

From Table 2 and Figure 3, a few observations can be made:

• Comprehension was the strongest component in both the baseline and midline programme assessment.• Strong comprehension of spoken instructions or questions does not necessarily indicate strong ability

to verbally respond to those instructions, as was evident during the baseline programme assessment;however, immersive language development activities can rapidly address such disparities.

• There was a considerable observed improvement in both the reading and speaking scores betweenthe baseline and endline programme assessment.

4.2 Aggregated Results, by School

Table 3 and Figure 4 look at the performance across each school. As with the previous section, thesevalues are not scaled according to the weights of each component, but rather, look at each componentindividually.

Comprehension Speaking Reading

Hombegowda (Baseline) 61 6.4 21

Hombegowda (Midline) 90 28 69

Jakkasandra (Baseline) 67 23 42

Jakkasandra (Midline) 85 71 67

KSRP (Baseline) 66 14 39

KSRP (Midline) 81 69 68

Satya Narayana School (Baseline) 53 13 11

Satya Narayana School (Midline) 80 48 43

Table 3: Baseline and Midline Performance Comparision by Schooland Component

From Table 3 and Figure 4, we can clearly observe improvements in all components at the midline stage.The performance in the speaking component at Hombegowda, while showing improvement between thebaseline and midline, is still at a much lower level than the performance at other schools.

4.3 Aggregated Results, by Class

Figure 5 shows the performance at baseline and midline according to class. As was seen in Table 1, theKaradi Path Magic English SLL programme was implemented in classes III through VIII. Although itis assumed that all students in the programme come from similar home environments, where access toEnglish and support with English are not commonly available, one should not assume that the childrenhave no exposure to English. This is particularly true in the case of the four schools which were part ofthis assessment since Samanvaya Foundation has had prior interventions in the area of trying to improvespoken English skills.

The data in Figure 5 reinforces the above concept. As can be seen, the children in the lower standardsperform at an overall lower level, both at the baseline and the midline. This can be indicative oftheir limited exposure to English within their environments. By comparison, at least in the area ofcomprehension, students in higher standards demonstrated a slightly higher starting point at the time ofthe baseline assessment, possibly indicating that they had started to pick up some English from whathappens in their environment.

8

Figure 4: Baseline and Midline Performance Comparison, by School and Component

9

Figure 5: Baseline and Midline Performance Comparison, by Class

10

4.4 Disaggregated Performance Comparison

In the following table, the prefixes “C”, “S”, and “R”, refer to “Comprehension”, “Speaking”, and“Reading”; the suffixes “BL” and “ML” refer to “Baseline” and “Midline”.

Overall, improvements were observed in almost all areas for all classes (Class 5 at KSRP is the singleexception). While the results are very promising for one year of implementation, there is still considerablescope for further development in speaking. This matches with the overall programme design, where thefirst level of the programme include strong elements to develop listening comprehension and readingability, with a stronger emphasis on Story Path activities in the second level of the programme.

School Class C_BL C_ML R_BL R_ML S_BL S_ML

Hombegowda VIII 12 18 8 28 5 22

Jakkasandra III 14 16 7 19 15 55

Jakkasandra IV 10 14 7 19 15 33

Jakkasandra V 13 14 1 11 8 54

Jakkasandra VI 13 17 26 32 16 49

Jakkasandra VII 13 18 15 29 14 59

Jakkasandra VIII 17 18 32 37 36 72

KSRP III 9 18 0.2 6 1 19

KSRP IV 10 13 3 11 9 30

KSRP V 16 15 18 37 12 62

KSRP VI 14 18 24 34 12 70

KSRP VII 14 18 24 36 15 74

Satya Narayana School III 9 16 3 15 4 30

Satya Narayana School IV 12 15 3 10 9 17

Satya Narayana School V 10 15 8 23 8 62

Satya Narayana School VI 11 17 3 21 22 38

Table 4: Disaggregated Performance Comparision

11

4.5 Histograms, by School

The histograms show the overall distribution across all the students at a given school. In all cases here, thecomprehension scores appear to be skewed to the left (more scores in the higher end of the distribution),indicating a higher level of comprehension. A similar strong skew to the left is noted in the speakingscores, which is a considerable improvement in what was observed in the baseline assessment. As withthe baseline, the reading scores do not demonstrate an obviously strongly discernible pattern, but doshow an improvement when compared to the baseline assessment.

While speaking scores showed a dramatic improvement, the maximum marks obtained was far below themaximum marks possible within the section.

The following pages include the following figures:

• Comparative histograms from the baseline and midline programme assessment.

– Comprehension (Figure 6)– Speaking (Figure 7)– Reading (Figure 8)– Total (Figure 9)

12

Figure 6: Histogram of Total Comprehension Score by School

13

Figure 7: Histogram of Total Speaking Score by School

14

Figure 8: Histogram of Total Reading Score by School

15

Figure 9: Histogram of Grand Total Score by School

16