WHAT IS EVALUATION?

Evaluation is assessing and judging the value of a

piece of work, an organisation or a service. Its main

purpose is to help an organisation reflect on what it

is trying to achieve, assessing how far it is

succeeding, and identify required changes.

Hutchinson & Waters (1987, pg: 97) stress:

“Evaluation is basically a matching process: matching

needs to available solutions.”

WE EVALUATE TO FIND OUT:

•what our program has done;

• how well it has contributed to the goal, met the

objectives and undertaken the strategies;

• what worked well and what didn’t, and why;

• whether there were any unintended outcomes; and

• what can be learnt from the program to improve

practice

and inform other programs.

OUTCOME EVALUTION

IMPACT EVALUTION

PROCESS EVALUTION

SUMMATIVE EVALUTION

FORMATIVE

EVALUATION

TYPES OF EVALUATION

FORMATIVE EVALUATION

• conducted early in the implementation of a

program;

• aims to identify problems that arise during

development and thus allowing modification.

SUMMATIVE EVALUATION

• conducted at the end of the course;

• focuses on effects or impacts;

• helps to decide what to do next.

PROCESS EVALUATION

• focuses on how the program has been implemented;

• assesses whether activities have been conducted as

planned.

IMPACT EVALUATION

• focuses on the immediate effects of the program;

• judges how well the objectives have been met.

OUTCOME EVALUATION

• focuses on the longer term effects of the program;

• judges how well the goal has been achieved.

EVALUATION: 

•Answers the question "How well did we do?"

(Qualitative evaluation)

•Answers the question "How much did we do?

(Quantitative evaluation)

Qualitative evaluation answers the questions below:

•What was learned?

•Are the learners using their new knowledge? If so,

how?

•What do the learners think about the lessons, the

teachers, and the materials?

•Do the students think that the courses made a

difference in their lives? If yes, what kind of difference?

•Did the teachers do a good job of communicating the

new information?

•Did the teachers respect and support the learners ?

•Do the learners like the materials?

•Do the students think the materials are appropriate

for each group of learners?

•Do the materials communicate information they want

to learn?

•Do the learners think that the program is successful?

•Which activities do they think are good? Which

activities do they think are not good?

•Do people in the community think the results of the

program are worth the cost and energy necessary to

get the program started and to keep it going?

Quantitative evaluation answers the questions below:

•How many people were in the target group?

•How many started the course? How many completed

the course? How many dropped out?

•How many books were produced?

•Are the test scores satisfactory?

•Does the course worth to the money spent on it?

LEARNER ASSESSMEN

T

COURSE EVALUATION

LEARNER ASSESSMENT

It helps to assess students’ performance either at the

beginning of the course or at the end.

COURSE EVALUATION

Course evaluation is the last, but not the least

important stage. It helps to determine whether the

course objectives are being met. Teachers should

evaluate their courses to improve and promote their

effectiveness.

Evaluation of the course is a brave step for the

teacher. He / she should be open-minded in hearing

and implementing learners' comments.

Although these two different evaluation forms assess

different things on the surface, we can not seperate

one from the other.

Evaluation of the learners does not only reflects the

learner’s performance but, to some extent, the

effectiveness or non-effectiveness of the the course

as well.

An ESP course is supposed to be successful in order to

enable particular learners to do particular things. If it

fails to meet its objectives, then something might be

wrong with the course design, the analysis of the

student competence or the methodology.

We do not expect learners to indicate where the exact

fault lies but this application will show us the possible

lacks of our programme.

Hence, it can be concluded as both course and learner

evaluation have a similar function in providing the

feedback for the ESP course; although each type has

other purposes and procedures.

Although having admitted the importance of

evaluation, there is a lack of guidance how the

objectives of the course be tested.

The lack of testing procedures can be attributed to

prejudices against the participating linguists and

other practitioners. Yet this does not imply that there

are no tests available.

For instance;

The London Chamber of Commerce and Industry

The Associated Examination Board (AEB)

Pitman Examinations Institute

Secretarial and Commercial

English

Cambridge University English for Business

and English for

ScienceThe City and Guilds of

London Institute

Technical English

What is lacking is the sound theoretical or emprical

basis for ESP testing.

There are three basic types of assessment:

•Placement Tests

•Achievement Tests

•Proficiency Tests

PLACEMENT TESTS:

This type of test is used to assess learners’ academic

ability in a certain field so that he or she can be placed

in the appropriate courses. It determines whether the

learner needs the course or what form of course the

learner should take.

If a learner is fully qualified in the required skills, no

further course is necessary. So it can be concluded as

placement tests sometimes act as proficiency tests.

A placement test is diagnostic as it indicates how far

the learner fails to meet the proficiency level. The

placement test has a formative value as the result of

the test used in forming the content and the nature of

the course.

How Can A Placement Test Be Trustworthy?

According to Alderson and Hughes (1981), there are

some doubts about the ability to diagnose learners’

needs accurately through tests. Nevertheless, in the

absence of no more accurate instruments, the course

designer has quite a few options. So these tests can

only be a guide for us. A good placement test should

indicate not only what the learner lacks, but also what

potential the learner has for learning.

ACHIEVEMENT TESTS:

Achievement tests are standardized measures of

knowledge, information, or procedural learning such

as how to do something. Achievement tests may

assess general academic skill areas, such as reading,

writing, or mathematics, or they may test for content

knowledge in a specific academic subject such as

biology or American history.

Achievement tests are used by school systems to

provide a standard measure of individual student

performance and provide an aggregated measure of

performance that enabling school systems to evaluate

their effectiveness. Achievement tests are also used as

part of the diagnostic assessment of individuals to

determine whether they have a learning disability, and

therefore qualify for special education services.

Basic principles that should be followed for

constructing an achievement test along with other

tests:

•Test what the students have learnt. This may not be

in exact correlation with what you have taught during

the lesson.

•Test what you actually want to test.

•Do not write test items demanding special knowledge

on one subject area.

PROFICIENCY TESTS:

It’s an exercise whose aims is to evaluate the

background of a person in a given branch of knowledge

with or without regard to specific academic learning.

According to Davies and West (1984), the primary

purpose of language testing is to assess whether the

students will be able to perform the language tasks

required of them. They say such tests are criterion-

referenced. With this application there is no fail / pass

distinction.

Instead, there is a scale of degrees of proficiency in the task.

Here is an example of a scale used for the British Council’s English Language Testing Service (ELTS).

1.NON-USER: Unable to use the language or doesn’t

provide relevant evidence of language competence

forassessment.

2. INTERMITTENT USER: No real communication possible

although single-word messages may be conveyed

and understood.

3.EXTREMELY LIMITED USER: Below level of functional

competence; although general meaning can be

conveyed and understood in simple situations; there

are repeated breakdowns in communication.

4.LIMITED USER: Basic functional competence limited

to familiar situations, but frequent problems in

understanding and fluency can make communication

a constant effort.

5.MODEST USER: Partian command of the language

coping with overall meaning in most situations

although some misunderstanding and lack of fluency

could block fluency.

6.COMPETENT USER: Generally effective command of

the language, although occasional misunderstanding

and lack of fluency could interfere with

communication.

7.GOOD USER: Operational command of the language;

occasional inaccurasies and misunderstandings in

some situations.

8.VERY GOOD USER: Fully operational command of the

language; occasional minor inaccurasies or

misunderstandings in some situations.

9. EXPERT USER: Fully operational command of the

language; appropriate,accurate and fluent with

complete understanding.

The proficiency tests requiring students to perform

certain tasks is closely related with the concept of ESP.

Proficiency tests for specific purposes should be able

to determine whether a student can perform the

required skills or not via their own topic of interests

rather than a general topic.

Although specific language proficiency tests are an

extension of the ESP principle, there are still some

problems about it. The problems are:

Proficiency tests are criterion-referenced. What should

the criteria be then? What skills and knowledge enable

someone to perform particular tasks?

Although there is a huge development in our

knowledge of language use, we still have lacks on what

makes communicative performance possible.

How specific is specific?

Can a test in Engineering be attributed to all branches

of Engineering hoping it to be a valid indicator?

So we should consider proficiency tests as a guide

until a more valid technique is found out.

As a result; the value of tests primarily depends on

how they are used. Both teachers and learners must

have a positive attitude towards them until they

disappear from our lives.

All the tests we have encountered should be seen as a

performance task in the teaching – learning process.

Even if a grade is given to a student, the importance

of the grade lies under its meaning.

Evaluation has two aims:

•assessment

•feedback

Although assessment can be seen as a means of

measuring what the learner knows, it also provides a

positive feedback to inform teachers and learners

about what is still unknown.

As an ESP course needs to satisfy a particular need,

evaluation helps to see how well the course is fulfilling

the need. It helps to determine whether it meets its

objectives. The result of this evaluation forms your

starting point for any necessary revisions, helps to

design your other related courses and guides to your

teaching methodology.

It plays a social role involving teachers, learners,

sponsors… etc. in evaluation. There are four basic

questions to be asked in course evaluation.

•What should be evaluated?

•How can ESP courses be evaluated?

•Who should be involved in the evaluation?

•When (how often) should evaluation take place?

WHAT SHOULD BE EVALUATED?

Your ability to collect information

&

Your ability to use information once it has been collected.

When considered ESP, the main aims are to meet the

learners’ needs as language learners and the learners’

needs as language users. The success of an ESP

course is achieved if the course satisfies both kinds of

needs. So we should ask the following questions:

Is the course fulfilling the learner’s language learning needs?

Has the course fulfilled

the learners’ language using needs?

Is the course fulfilling

If you get a YES to any of these questions, then just

go ahead! You are on the right way!

However, assuming that you might have only an

ORDINARY YES instead of an outright NO, the next

question should be asked.

What areas of need are not being / have not been

fulfilled?

If we know where the problematic areas are, we can

try to find out the source(s) of the problems. Here are

some other relevant questions on what to evaluate on

course evaluation.

• Were the unfulfilled needs identified during the course design project? If not, why not?

• How can the course design process be improved to avoid this problem in future?

• Taking these needs into account, how can the course be changed?

• If the needs were identified during the course design process, is the fault in:

a. the syllabus(es),

b.the applied materials,

c. the teaching and learning techniques,

d.the testing procedures,

e. logistical / administrative arrangements,

f. the course evaluation system?

HOW CAN ESP COURSES BE EVALUATED?

Evaluation can be done in two different ways: implicitly

and explicitly. Implicit evaluation takes place during

the semester; students’ grades, participation, and

motivation give clues to the teacher on how their

learning is going on. Explicit evaluation may take place

at the end of the course or after students have

experienced it. Using questionnaires, surveys, talks,

etc. teachers ask the students to express their attitude

towards the subject matter, instructional methods,

activities, teacher's role and so on.

Although there are lots of techniques in evaluating

ESP course, most common ones are the followings. It

is not necessary to use only one of the following

techniques at a time. More than one technique can

also be used at the same time.

•TEST RESULTS

•QUESTIONNAIRES

•DISCUSSIONS

•INTERVIEWS

There is not a strict limit on which technique you

should choose. It depends on your own situation.

Yet having got the evaluation information is only the

beginning in this process. The information you got

must be gathered together and needs to be discussed

with all the participant parties (teacher – learner –

administrator – sponsor…etc) and some conclusions be

drawn.

This course evaluation report should be used as a

basis for decision making. They are valuable

suggestions to redesign your course and correct the

problems.

EVALUATION INVOLVES

ASKING QUESTIONS

ANSWERING QUESTIONS

ANALYSING, REFLECTING ON & JUDGING THE

EVIDENCE

MAKE CHANGES!

WHO SHOULD BE INVOLVED IN THE EVALUATION?

Whoever will involve in the evaluation may vary, but

the very closely related participants will be the ESP

teachers, the learners, and the course sponsors as

well as the former students.

Evaluation is primarily concerned with people’s

perceptions of value; therefore their evaluation may

also vary.

The teacher can have difficulty in getting the exact

view as the students may be reluctant to criticise

authority for fear of their own assessment.

The students can also be indifferent thinking that, this

evaluation will be beneficial to the future students but

them.

However, according to Waters (1985), the best basis

for promoting frank and useful feedback can only be

achieved through an open and trusting relationship.

Who you ask and how you ask also affect the results.

That’s why, the methodology of evaluation process

should be taken into a great consideration. You should

explicitly note how you obtain the information, from

whom you got it and by what means.

WHEN & HOW OFTEN SHOULD EVALUATION TAKE

PLACE?

It is difficult to mention how frequently an evaluation

should take place. Whether we apply this experience

too often or rare, there exists danger in both.

The most important frequencies of the evaluation

should be:

•in the first week of the course

•at regular intervals throughout the course

•at the end of the course

•after the course (if possible)

The evaluation made after the class is potentially the

most valuable one as the learners will be in a position

to judge how well the course was prepared for the

target situation they are in now.

•Evaluation can be time-consuming, complex and

frustrating.

•There might be no obvious solution to the problem.

•Feedback from one party may contradict feedback

from another.

•Learners might take it as an incompetence.

•Criticising a course might be seen as a lack of respect

to the authority.

STUDENT ANALYSIS

FORMULATION OF GOALS AND OBJECTIVES

CONTENT

SELECTION OF TEACHING MATERIALS

PLANNING THE COURSE

COURSE EVALUATION

REFERENCES

•Hutchinson, T. & Waters, A. (1987). English For Specific Purposes: a learning-centered appproach. Cambridge: Cambridge University Press .

•Spencer, L.(2003). “Quality in Qualitive Evaluation”: A framework for assessing research evidence.

•The Brazilian ESP Project. Ed. Holmes, J. (1988)

•Cañado, María Luisa Pérez & Esteban, Ana Almagro. (2005). Authenticity in the Teaching of ESP: “An Evaluation Proposal”