Evaluating Evaluating Multilingual Multilingual

Education ProgramsEducation Programs Seminar on Multilingual Education

Kabul, 13-18 March 2010 Dennis Malone, Ph.D.

Preliminaryresearch

Advocacy & mobilization Recruitment

Orthography development

Graded reading materials

Training & supervision

Curriculum

Supportive political environment Components of

successfulMLE programs

Management & coordination

Documentation &evaluation

Funding

ParticipatorParticipatory y

evaluation evaluation in MLE in MLE

programsprograms

1) Focus

What specific part of the program do

we need to evaluate?

2) Purpose

Why do we need to evaluate this part?

What will we do with what we learn about

it?

4) Information sources

Where will we get the information

we need?

3) Indicators

What do we want to learn? How will we

know if we have achieved our outcomes & outputs?

5) Methods / Tools

How will we get the information we

need?

7) Time frame

When will we begin the evaluation?

When will we finish collecting

information? When will we finish analyzing the

information? When will we finish

documenting what we have learned?

6) Responsible persons

Who will be responsible for collecting and analyzing the

information and documenting what has been

learned?

Two Evaluation TypesFormative

Evaluation done during the program

Purpose: to identify strengths and weaknesses

Leads to actions that maintain what is strong and strengthen what is weak.

Summative

Evaluation done at the end of the program

Purpose: to determine if the program goals and outcomes were achieved

Provides relevant information to program funders and other stakeholders

Two Evaluation Approaches

QualitativeWhy did it

happen?How did it

happen?Who made it

happen?

QuantitativeWhat happened?

How much happened?

Compared to what?

•Teachers’ performance

•Quality of curriculum and of reading materials

•Achievement of program goals

•Costs of the program compared to stakeholders’ satisfaction

Evaluations that focus on the quality of the program assess:

•Focus group discussions

•Interviews (keep accurate quotes)

•Observation

•Portfolios

•Informal discussions

•Examination of materials that were produced

Evaluation methods that focus on quality:

• Achievement of outcomes and outputs

• Learners’ progress in achieving reading and writing skills

• Costs of the program compared to achievement of program outcomes and outputs.

• Costs of the program compared to learners’ progress through the system

Evaluation methods that focus on quantity:

•Questionnaires (short answer)

•Tests/Examinations

•Analysis of attendance records, student progress reports, other documents

Evaluation methods that focus on quantity

What information would you need to answer the questions on the next slide?

What assessment tools (questionnaires, interviews, etc.) would you use to get the information?

Are the Grade 1 parents satisfied with the MLE program?

How much progress did the Grade 1 children make in reading and writing in their language?

How good are the children’s Stage 1 reading materials?

Are people who completed the adult literacy course two (2) years ago reading and writing now? What do they read and why?

How well has the program achieved the goals, outcomes and outputs of MLE program plan (for report to funders)?

Planning the evaluation process

Program Outcomes

Indicators Data needed

Methods / instrumen

ts

Person(s) responsibl

e

Time Frame

   

         

   

         

   

         

   

         

 

Adapted from Chong project, Thailand

Evaluating reading materials—some

suggestions

Do action research with Grade 2 or 3 students to see if they can read Grade 1 materials easily?

Do “cloze” tests with Grade 1 children using Grade 1 reading materials.

Analyze reading materials in Grade 1 to see which are “used” most, which least? (Well-used, dirty, wrinkled, torn; OR rarely used, new-looking, clean, unwrinkled.)

Interview children about their favorite books.

Documentation

Results of preliminary research Reports Records Documents Assessment Results

Results of Preliminary Research

Situation before program begins Language attitudes literacy uses

Education records from previous years (in formal education—for children’s programs) to use for comparison

Community’s identified goals and needs

Context analysis (key factors)

Reports Training activities Supervisory visits Public relations activities (e.g., school

opening ceremony) Support committee meetings Meetings with donors

Records Training activities Public relations activities (e.g., school

opening ceremony) Support committee meetings Meetings with donors Budgets / donor reports

Documents

Copies of all curricula and instructional materials

Samples of reading materials Samples of the students’ work Copies of all assessment instruments

including exams

Assessment Results…

of pre-tests of follow-up assessments of learners’

progress in formal or non-formal education

of periodic evaluations of the community’s perception of the program

of end-of-year evaluations of materials, training, curriculum

Questions for group discussions

What kind of evaluations are needed for programs in your community?

Who can do the needed evaluation? What kind of training is needed for

implementing the evaluation? What are challenges to effective

evaluation? What are solutions?