SECTION ONE

EXPLORATION OF THE STUDY AREA

1.1 INTRODUCTION AND BACKGROUND

In the previous South African education dispensation, prior to the democratic

elections of 1994, educators were not given the opportunity to participate in the

process of evaluating and selecting Learning and Teaching Support Materials

(LTSM) for their classrooms (Gauteng Department of Education LTSM Policy

2002:1) This resulted in selection of inappropriate LTSM or materials that were

not relevant to the school community. This was in violation of the educators’ right

to having a voice in the selection of materials and making a contribution to

education as stipulated in the United Nations High Commission for Human

Rights (UNHCHR), article 26 of the Universal Declaration of Human Rights. This

is particularly because parents as well as teachers have, through community-

based forums, expressed their wish to be included in order to participate in the

process (Monyokolo 1993:9).

In South Africa, evaluation of Learning and Teaching Support Materials (LTSM)

is an important function for every provincial education department. This

includes the Gauteng Department of Education (GDE), which is the focus of this

study. This study therefore will inquire into how educators as evaluators engage

with the instrument that is used to evaluate LTSM. An endeavour is made to

detect problems and successes that educators as evaluators experience in the

process and whether such an evaluation instrument is effective, increases

efficiency and has impact.

In the new democratic dispensation in South Africa, educators are regarded as

equal, active participants in the process of selecting LTSM. They are

encouraged to work cooperatively with their peers. They are given the status of

what Dobson (2000:206) refers to as internal evaluators of LTSM. The inclusion

1

of educators in this process is also in line with democracy as borne in the

National Curriculum Statements (RNCS) and the South African Schools Act of

1996. This is also an endeavour to fulfil the obligation of the GDE to ensure that

materials are carefully and professionally selected for learners and educators, as

stated in the Department of Education (DoE) policy document (2002:3).

In the GDE, the process discussed above is managed by the LTSM Sub-

directorate. Ideally the LTSM evaluation and selection process should take

place 18 months prior to the date scheduled for the delivery of LTSM to schools.

This is particularly important because the intention of the GDE is that all learners

will have LTSM by the first day of the school year.

To achieve this obligation, the GDE designed and utilised a National Curriculum

Statement compliant evaluation instrument for the Foundation and Intermediate

Phase (See Addendum A). As stated in the foreword of the GDE evaluation

instrument (2003:3), the instrument is used on a continuous basis for evaluating

LTSM at various levels (e.g. Provincial, school, phase and grade levels). The

criteria stipulated in the evaluation instrument and their utilization are what the

GDE demands from educators and other stakeholders and these may be

referred to as professional demands or bureaucratic conceptions (Scriven in

Kellaghan, Stufflebeam and Wingate 2003:17). Furthermore, there is an

endeavour to give the educators an opportunity to test for the validity of the

evaluation instrument and to make value judgments. This is in compliance with

what Scriven (in Kellaghan et al., 2003:16) classifies as normative theories.

The evaluation instrument is Phase and Grade specific. The front cover of the

instrument enquires about the evaluators who will actively participate in the

process of LTSM evaluation, details on the LTSM (e.g. Title, grade, ISBN,

publisher) and the identification of the instrument (its name and Phase/ Grade).

The first page of the instrument is a Foreword by the GDE. In this section, an

attempt is made to introduce the process and procedures to be followed during

evaluation as well as the intended product, the Review Guide or catalogue.

2

Coupled with the introduction to procedure to evaluate, are the notes to the

evaluator as well as the lay out of the sections and divisions of the actual

instrument.

As LTSM head in the GDE, I am involved in the evaluation and selection of

Learning and Teaching Support Materials (LTSM) for the schools. The task of

LTSM evaluation outlined above is accompanied by activities such as the training

of educators on the same and related tasks, appointment of tenderers and

service providers and procurement of LTSM for the schools. As the project

manager, I have always been concerned about the effectiveness of the

instrument used for LTSM evaluation.

1.2 STATEMENT OF THE PROBLEM

The impracticality of evaluators of materials performing the task of LTSM

evaluation without a relevant instrument is a problem. To attend to this problem,

the GDE responded by providing an evaluation instrument to be used by

evaluators at different levels. However, the main concern should be whether or

not the instrument addresses the needs or reaches the intended outcomes. It is

also important to remember that the main intended outcome is for the educator

to be able to evaluate and select appropriate LTSM that are usable in the

classroom.

The above statement is an indication of the reasons for this inquiry to take place.

This study intends to guide educators as evaluators on how to engage with the

evaluation instrument. This will, therefore, assist in establishing the usefulness

of the evaluation instrument. The findings of this study could assist the GDE in

deciding whether the evaluation instrument they are presently using should be

adapted, adjusted, maintained or completely discarded and replaced by

something else.

3

Based on this background, the research question for this study is: Does theGDE instrument for LTSM evaluation increase effectiveness, efficiency andimpact in the evaluation, selection and utilization of materials?

1.3 AIMS OF THE STUDY

The main aim of the study is to find whether the GDE evaluation instrument

effectively makes an impact on the LTSM selection process, which could

therefore lead to efficiency among all the participants.

The sub-aims of the study include knowing if:

• evaluators can use the instrument independently

• the educators are willing to do evaluation as part of their daily routine

• the instrument encourages collaboration among educators

• the instrument is regarded as a core element for LTSM evaluation

• The instrument does empower educators to select only appropriate

LTSM.

1.4 RESEARCH DESIGN AND METHODOLOGY

To make this inquiry possible, the researcher adopts a predominantly qualitative

approach with a minimum slant toward quantitative research. In qualitative

research, as opposed to quantitative, the researcher is able to obtain more detail

and a larger variety of perspectives to the issues that are being investigated.

Henning, Van Rensburg and Smit (2004:6), describe this participative approach

as a more open-ended way in which the participants give their opinions, view

points and also demonstrate their actions. In other words, the variables are not

controlled. Melody (2002:148) emphasises that the participants are equally

human and should be given that status during inquiry. Furthermore, this type of

research allows all the participants (researcher and respondents) the liberty to

probe issues deeper as discussions progress.

4

In quantitative research the variables are controlled. The instruments are

designed to control the variables, for instance the respondents are not allowed to

express their viewpoints over and above what is contained in the questions on

the research instruments. Apart from causing frustration among the

respondents, the research inquiry becomes limited by what is contained in the

instruments. To avoid this situation, in this inquiry the researcher uses both

qualitative and quantitative to enable the respondents to participate in

discussions and clarify their respondents or even significant information that may

have been omitted. The research methodology, however, attempts to allow for

looking at data, finding the meaning of data, interpreting the meaning of data and

applying data in the best suitable manner (Henning et al.,, Van Rensburg and

Smit, 2004:4).

Furthermore, in the following section the literature on materials evaluation is

reviewed. Thereafter, selection of a specific research methodology is made with

the elaboration on the way data will be analysed resulting in a discussion of the

findings. A literature review gives the theoretical background in which various

authors and specialists give their perspective on the evaluation of learning

materials, and relevant theories such as radical and social constructivism and

transformative learning. In the research methodology section, there is

elaboration on the data, identification of the population and sample as well as

criteria for selection of materials’ evaluators.

In addition, I analyse the context within which the research takes place and also

described the sampling method I used. However, this section would not be

complete without the discussion on data collection methods in which I use survey

questionnaires and focus group interviews. I also give an exposition of how I

engage in the exercise to measure whether or not the questionnaire is valid and

reliable. The findings of the research are only useful if they are analysed and

made ready for implementation purposes. To analyse the data, I employ the

constant comparative method as presented by Maykut and Morehouse (1994).

5

The data collection methods used are survey questionnaires and focus group

interviews (considering their validity and reliability). After that, I present the

findings and perform data analysis to back up my conclusion and make

recommendations.

1.5 CONCEPT CLARIFICATION

The above-stated research design and methodology, in this inquiry, are

implemented to assist the GDE in the possible best way to evaluate LTSM.

Later on, in this section, the rationale for the inquiry is explained. However, prior

to delving into the rationale and theoretical orientation, it is important to clarify

what LTSM are and also to explore their different categories and types. The

National Department of Education (DoE) (2002:3), in its LTSM policy document,

adopted the term “Learning and Teaching Support Materials” (LTSM). Prior to

that, the term had been “Learning or Learner Support Materials” (LSM). The

two-fold naming of LSM was based on the fact that Departmental officials argued

whether it should be ‘Learner’ or ‘Learning’ Support Materials. Some officials

thought because it is intended to support the learner, it should carry the prefix

‘Learner’ (GDE 2003:3).

In addition, policy also stipulates that LSM was intended to support the learning

process undergone by the child, and should therefore carry the prefix ‘Learning’

(GDE 2003:3). This poses a challenge to the social constructivists who view

learning as a social collaborative process which happens between an individual

learner and others (co-learners or community). This implies that learning

happens in a dynamic social context (Gravett 2001:20). However, what is of

essence is that the learner actively constructs knowledge in the learning process,

in which his/her existing knowledge and experience play a crucial role. This

implies that the acronym LSM can harmlessly be utilized for both the concepts.

6

The GDE and Gauteng Institution for Educational Development (GIED) (1999:2)

stated that the term ‘Learning Support Materials’ refers to any kind of materials

that can be used to assist the learner to learn or facilitates learning. ‘Teaching

Support Materials’ (TSM) refers to materials that assist the teacher to teach. The

justification for the inclusion of the name ‘teachers’ in the term LTSM was that

teachers felt they were neglected and not given much support in terms of

teaching materials. The DoE decided to allocate some funds for Teacher

Support Materials. For the purpose of this study, I will use the collective term

‘LTSM’, since it encompasses both TSM and LSM.

The collective LTSM includes printed media, stationery and other materials.

Printed media include, among other things, textbooks, educators’ guides,

learners’ guides, learner workbooks, readers, atlases, dictionaries, magazines,

newspapers, charts and posters. Stationery includes exercise books, drawing

books, examination books, map books, paper glue, crayons, pencils, pens, fibre

pens, marking pens, news print, duplicating paper (not for office but for learners)

and erasers. ‘Other Materials’ include materials falling outside the

aforementioned categorization like educational equipment such as science kits,

puzzles, laboratory equipment, veniers, scales, tape measures, puzzles and

games. It also includes consumables such as soap, groceries for Home

Economics, some cartridges and glue (GIED/GDE 1999:2).

The above clarification is made to alert the reader that LTSM is not limited to

textbooks only and that an evaluation instrument needs to have sections for

evaluating a wide variety of LTSM. Other types of LTSM, such as educators’

guides, learners’ guides and workbooks, may be regarded as forms of textbooks.

As specified by Davies (2000:126), textbooks, in whatever form, are aimed at

fulfilling three main functions, namely ‘interpersonal’ (relationship between writer

and reader), ‘ideational’ (selection of information) and ‘textual’ (constructing a

coherent message). By means of appropriate instruments, evaluators

endeavour to establish whether these functions are fulfilled by LTSM.

7

1.6 OUTLINE OF THE REPORT

This work consists of five sections each dealing with a particular aspect. The

outline of the sections is as follows:

Section One introduces the research project to the reader. The introduction

provides a background in which the reader is given the context and focus of the

study, in particular the history of evaluation and selection in the GDE (that is

located in South Africa). The problem that necessitates this research is stated,

followed by the aim of the study. There is also a description of the research

design and method. With regard to concept clarification, the main terms that are

used in the essay are clarified.

Section Two consists of a Literature Review, in which is explored some of the

texts on evaluation and selection of materials, and evaluation of project

effectiveness, efficiency and impact. The first part deals with the different views

on definition while the second part is on effectiveness, efficiency and impact of

projects. This is followed by an exploration of reasons for evaluation to take

place. Different evaluation models, as presented by Boulmetics and Dutwin

(2000) are discussed.

Section Three first discusses research methodology in the form of population,

sample and data collected for analysis. This is coupled with research orientation

and context in which the scene is set. The survey questionnaire (including test

for validity and reliability) and focus group interviews that I used to collect data

are fully discussed. This discussion is not complete without the description of

the interview schedule,

Section Four mainly focuses on data analysis and research findings. Actual

data analysis takes place. The findings are discussed, grouped into themes

(using Maykut & Morehouse, 1994:127), and a constant comparative method is

presented. These findings are matched against the reviewed literature.

8

Section Five concludes the study and recommendations are made. In this

section, conclusions are drawn based on the findings. Finally, recommendations

are made to the GDE on areas that they might need to make improvements on.

1.7 SUMMARY

In this section, the overall intention of this inquiry was introduced. I explored the

different definitions of LTSM, the key concept in this inquiry. Furthermore, I gave

the background and history of the evaluation of LTSM starting from a general

perspective and narrowed down to the GDE context. This is because what

happens in the world directly has a bearing on what happens in a small area or

locality. In the last parts of this section, I state the problem which is a focus of

this study. I would like to re-iterate the aim of this inquiry which is to evaluate the

effectiveness, efficiency and impact of the evaluation instrument used by the

GDE to test whether LTSM are appropriate for use in the classroom. The next

section focuses on the Literature Review which deals with the exposition of what

various texts on evaluation specify.

SECTION TWO

9

LITERATURE REVIEW

2.1 INTRODUCTION AND DEFINITION OF THE CONCEPT EVALUATION

In the previous section, it has been argued that the prominent aspect on which

this work is based is the evaluation of an evaluation instrument. This therefore

suggests that prior to engaging with the actual exercise, it is mandatory to

explore the works of different writers and theorists on basic evaluation. In so

doing, I shall, at all times endeavour to apply the knowledge in the exercise of

evaluating the instrument used by GDE to evaluate LTSM. Some of the

knowledge needs to be considered prior, during and after an evaluation session

or project. It should also be noted that evaluation is a process that requires one

to construct, test and utilize certain special artefacts, for instance an evaluation

instrument. Evaluation exercises, as suggested by Boulmetics and Dutwin

(2000:2) happen mostly in controlled environments.

This section of my work consists of four sub-sections that deal with literature

review in some detail. In Sub-section 2.1, I introduce and define the generic

concept of evaluation and later on attempt to narrow it down specifically to the

evaluation of LTSM. Definitions by various educationists and evaluation

theorists are explored and finally fused to form an understanding of evaluation of

LTSM. In sub-section 2.2, I analyze deeply three main concepts that are

centered on evaluation, namely: efficiency, effectiveness and impact. These

concepts also serve the primary basis for conducting an evaluation. Hence, they

connect this sub-section with the next one (Sub-section 2.3) which specifically

deals with formal reasons for evaluation as put by some evaluation experts.

Section 2.4 explores these evaluation models: discrepancy, goal-free,

transaction, decision and goal-based model and the criteria used for evaluating

LTSM. Such criteria are primarily to test whether standards are adhered to.

10

These standards include compliance with the curriculum requirements (GDE

LOTEA policy 2003: 2).

In view of the above, the starting point will be to explore the definition of the

concept ‘evaluation’. Boulmetics and Dutwin (2000:4) define evaluation as a

systemic process of collecting and analyzing data in order to determine whether

and to what degree objectives are achieved. Furthermore, Boulmetics and

Dutwin (2000:4) view evaluation as a systemic process of collecting and

analyzing data in order to make informed decisions for future use. Evaluation is

perceived as a social process that is different from development and which

might, if not carefully employed lead to a ‘’we- they’’ polarity. It is therefore upon

education departments to ensure that none of the stakeholders feel excluded

from the process. (Burtman & Fletcher, in Borich 1974:39).

It is quite evident that an education department that strives for unity, like the

GDE, will at all costs avoid a situation that may arise with a ‘we-they’ polarity. A

department would like its employees to feel like they are part of the same

homogeneous group. In that set up, evaluation forms part of what may also be

described as identification of discrepancies between where a project is and

where its leaders would like to be (an indicator for the real and ideal situations).

Furthermore, a project should have the qualities of engagingness, accessibility

and generalizability. These constructs are also critical in understanding the

support that texts must provide children (Menton & Hiebert 1999: 07).

Education departments need to have measures to ensure the absolute objectivity

of evaluation. This would assist in producing results that are acceptable to

publishers who are stakeholders in education and more importantly, businesses,

who need to sell their LTSM (Denning (1992:14). To be more precise, the

evaluation of LTSM entails objective and intensive reading, bearing in mind

criteria as stipulated in an evaluation instrument; writing down analytic points

(strong and weak) to determine whether the LTSM are educationally sound,

11

scoring and writing qualitative reports/ reviews about every component of a

publisher’s LTSM (GDE 2003: 1).

These definitions suggest that when an organization engages in an evaluation

the primary purpose is to see rapid change, growth and development. Borich

(1974:27) juxtaposes educational evaluation with the cost of living stating that

both have one thing in common, namely a propensity for rapid change. The

GDE therefore, needs to be dynamic to keep up with the demands of the

publishing world and the curriculum.

The concept ‘evaluate’ together with other related or synonymous concepts

have a common goal. This refers to words like ‘assess’, ‘appraise’, ‘weigh’,

‘consider’,’ review’ ‘gauge’, ‘calculate’ ‘moderate’, ‘adjudicate’ ‘referee’ ‘umpire’

and ‘estimate’. The common goal or intention implied in the aforementioned

terms is to find the value, worth or usefulness of a subject. This is what

evaluators of materials are conscious of when evaluating LTSM.

Based on the above definitions, one can assert that an effective evaluation

process is one that operates within certain parameters or criteria and its planning

is based on a specific purpose, intended outcomes and objectives. Furthermore,

the process of evaluation is scientific and is driven by data collection and data

analysis. The reasons for the effort of collecting and analyzing data are different

and reflect a notable difference in philosophies (Boulmetics & Dutwin 2000:4).

Evaluation, review and selection of LTSM are interdependent processes and

serve as a pre-requisite for LTSM procurement. Depending on a number of

possible causative factors, the GDE possesses the right to use its prerogative in

deciding whether or not to undertake any one of the three steps. However, the

definition below is meant for the purpose of indicating the common

understanding of the concepts among all users and participants within the GDE

(GDE LOETA policy 2003:3).

12

For the purpose of this inquiry, data was collected to examine the effectiveness

of the evaluation instrument used by GDE to evaluate LTSM. The primary focus

in this exercise is establishing the veracity of the instrument so as to see if it is

worth retaining as is or needs to be adjusted to meet the requirements of the

GDE. This implies that evaluation makes a claim of the value of the instrument

in relation to the overall process of LTSM evaluation. As Boulmetics and Dutwin

(2000:5) note:

Many experts agree that an evaluation should only assess program results

(whether objective and intended outcomes were met) but also identify ways

to improve the program

The evaluator, a human being, is central to the evaluation process. Project

leaders, according to Borich (1974:27), should always bear it in mind that one of

the most significant developments in the intended rapid growth referred to above

is the human and personal nature of the evaluator. The evaluator deals with a

variety of people like other evaluators, publishers, colleagues and project

leaders. In addition to these relationships, the evaluator also needs to engage

with his/her artefacts in order to be able to measure the LTSM successfully.

The role of the evaluator, in this context is for good and not for ill – to protect the

end users from purchasing inappropriate LTSM and to assist the publishers on

improving their LTSM to the required standards. Borich (1974:101) summarizes

the role of the evaluators as: to demonstrate that the materials developer has

been successful; that the materials meet the requirements and intended

outcomes; and that the overall quality of the materials is to the required standard.

One may therefore conclude that the objective for LTSM Evaluation and Review

is to make a positive impact on the effectiveness and efficiency of its

provisioning, defined as having the right materials at the least cost in the right

institutions at the right time (Eastern Cape Department of Education, 2004).

13

In order to perform the task of evaluation of LTSM successfully, the evaluator

needs to be given preset criteria so as to ascertain the parameters within which

to function. The criteria are set by the relevant education department, in this

regard the Gauteng Department of Education (GDE) through its LTSM unit. The

criteria are enumerated in a grade or phase – specific evaluation instrument that

is adapted to the needs of the curriculum, the Revised National Curriculum

Statement (GDE 2003:2).

Upon completion of the evaluation process, education departments select LTSM

that meet specified minimum requirements and place those in a list of approved

LTSM. The list is eventually converted into a catalogue which the GDE names

the ‘List of Approved Materials’. It is important to state that evaluation occurs at

different levels: outside the school at the Evaluation Centre and at the school by

individuals and teacher teams (e.g. LTSM committees within the school). This

implies that schools have a role to play in the selection of LTSM considering the

needs of their teachers, learners and the surrounding school communities (GDE

2003:2).

However, it is important to note that evaluation of LTSM is not the only activity in

an education departments’ process map (see Addendum C) but forms an

important and indispensable area. This is evident in Figure 1 and 2 as specified

by models used by the Eastern Cape Department of Education and GDE

respectively (GDE 2003:3).

2.2 PERFORMANCE OF EVALUATION TO MEASURE EFFICIENCY,EFFECTIVENESS AND IMPACT

Public and private enterprises are faced with the task to lead, monitor and control

projects and programmes. Likewise, the GDE is also faced with such a task.

The LTSM evaluation and selection process is one of the biggest projects within

the GDE. In ensuring that the project succeeds and is sustainable, a project

leader needs to be a good decision maker. This entails assessing the project’s

14

efficiency, effectiveness and impact. If the results prove to be negative, the

project leader should consider either abandoning the project or introducing new

alternative justifiable measures to rescue the project.

To achieve the aforementioned, as reflected on its LTSM evaluation policy

known as Limited Open Equivalent to Approval (LOETA) (Addendum E), the

GDE shows its participative approach and has certain demands towards its

officials, educators, evaluators and learners concerning criteria and standards for

LTSM evaluation and selection. To accomplish that, criteria for evaluation and

selection are laid down and expressed through the evaluation instrument. This is

what the GDE does not negotiate with its stakeholders but states as a

requirement. Scriven (in Kellaghan, Stufflebeam & Wingate 2003:17) refers to

similar requirements as professional demands or bureaucratic conceptions.

While employing the above, it is however important to note that an education

department should not intend to use professional demands to limit the educators’

mobility or liberality but should aim to display a pattern and to set parameters

within which they should work to benefit the process of teaching and learning.

To meet this end, in this study, there is an endeavour to give the educators an

opportunity to test for validity of the evaluation instrument and to do value

judgment. This is in compliance with what Scriven (in Kellaghan et al., 2003:16)

classifies as Normative theories. The implication is that what evaluators do is

driven by the current swings of the different sections of society or stakeholders

like the public, their profession and their bureaucratic conceptions and

convictions. It could be outcome-oriented evaluation. In other words when

impact is calculated the outcomes are considered.

A professional evaluator, according to Scriven (in Kellaghan et al., 2003: 17), is

expected to do more than just reporting facts as they are. He/she should also

critique the findings and synthesize the results to measure the usefulness of the

process and of the instrument that is used to evaluate LTSM. By so doing an

15

evaluators’ team will end up with an evaluative conclusion about the merit of the

evaluation programme and as an outcome of the evaluation of LTSM.

The professional demands of the GDE suggest that the evaluator is bound by

the predetermined standards as decided by the curriculum and reflected on the

instrument for evaluation. For effectiveness, efficiency and impact, the way in

which the GDE conducts, this process should neither be carefree nor goal free.

Scriven (in Borich 1974:30) describes goal-free evaluation (GFE) as the one in

which the evaluator performs the function from a more general perspective.

Although the GDE would allow the evaluator to contribute existing knowledge

and skills, the evaluator is expected to follow the guidelines as specified on the

instrument for evaluation. The question for this inquiry is whether the instrument

is reliable and valid.

As outlined in section one of this work, educators and other stakeholders are

given the status of equal partners in education. This means that if changes are

to be implemented in the evaluation system, it should make sense to them. For

instance, a teacher and his/her class are a sub-system within a school but

changes cannot be forced upon them or else the changes will be resisted. This

implies that for the GDE to implement changes, the professional, the educator

needs to be consulted more particularly in the planning stages of the evaluation

process.

Effectiveness can also be measured against the selected LTSM themselves.

Serguin (1989:51) states that learning materials, in particular school textbooks,

are one of the factors that determine the effectiveness of teaching and learning

in school and the yield of educational systems. This implies that the quality and

relevance to the intended outcomes of LTSM are of significance. These qualities

should be carried by manuscripts that are presented to the education department

for evaluation. Content, pedagogical approach, language and illustration are

some of the determinant factors for textbook effectiveness (Potenza in Kromberg

et al., 1993:48).

16

The aforementioned still implies that educators are involved in a dialogic

construction of knowledge with their fellow educators (Gravett 2001:20). They

engage in discussions on the instrument and work in teams at the Evaluation

Centres and back at school as they have to co-operate with their School LTSM

Committees that the GDE established in the years 1997/8. Lincoln (2003:51)

refers to this knowledge as a socially constructed process. This is because the

educators regard themselves as a community of ‘knowers’. The professional

demands lead them to engage in participatory work. Therefore, in this work they

are referred to as ‘participatory evaluators’.

Boulmetics and Dutwin (2000:5) collectively term the concepts of efficiency,

effectiveness and impact as levels of programme evaluation. Efficiency refers to

the analysis of the costs. Costs may manifest themselves in rand amounts, the

number and variety of people involved in the project, time spent on the project,

facilities required for the project to run efficiently and materials or resources. An

evaluation instrument needs to measure whether processes are sufficient, cost

effective and correctly sequenced. It should also test whether system

arrangements support effective and efficient supply and prevent leakages

(Eastern Cape Education 2004: 17).

Effectiveness examines whether the planned project activities do what they were

supposed to do. In the case of the GDE, during LTSM evaluation, a scene is set

where there are teams of evaluators who hope to achieve selection of

appropriate LTSM as their intended outcome. Effectiveness would measure

whether the instrument does enable the evaluators to competently select only

appropriate LTSM. In other words, effectiveness is measured in substantive

changes in knowledge, skills, and attitudes. It is a way to assess whether the

activities provide the skills to run the project now and in future.

Impact is the most important evaluation consideration as it refers to attainment

and non-attainment of outcomes (Borich 1974:88) It examines whether there are

17

long-term and sustained changes (the extent) in a particular target population. In

the GDE evaluation process, if there is an impact, it will manifest itself as a

change among the evaluators and also within the evaluation process itself (the

decision making process whether to accept or reject a title that is being

evaluated). This study further sets out to find whether there are any desired

changes and whether the introduction of the evaluation instrument changed the

lives of evaluators and if it increases satisfaction among evaluators or not.

Based on the above, one may argue that the GDE is faced with the task of

measuring the impact of its LTSM evaluation project. This is purely because the

task of evaluating LTSM is extremely significant as it is a means to ensure that

the design features of the Revised National Curriculum Statement (viz: Critical

Outcomes, Assessment Standards Learning Outcomes) are achieved. This

would assist in ensuring that South Africa ends up with competitive, world-class

citizens.

To complement the above Boulmetics and Dutwin (2000:7) bring in another

perspective as they assert that the challenge to assessing impact is that it

manifests itself over time. The project managers may have resigned from their

employment when the results of impact are observed. My view is that this

statement is not completely true as part of the impact can be observed within a

short period. For instance, upon orientating evaluators, in a matter of hours they

commence with the evaluation activities. It is however difficult to measure

whether they are fully enabled by the orientation. As a preparation for the

assessment of impact, data may be collected by observing learner turnover – in

this case the evaluators’ turnover; circumstances at the schools after evaluation

orientation and activities and also by measuring change and how resources are

utilized. This is done to enable the project managers to identify areas of

improvement.

2.3 FORMAL METHODS FOR EVALUATING LTSM

18

The aforementioned suggests that evaluation is a specialized complex field of

work that may be accompanied by specific activities. These qualities of the

evaluation process suggest that in order to perform a reliable valid evaluation

activity, evaluators should be carefully selected and orientated and the

instrument for evaluation has to be developed in a manner that will enable it to

have as minimal error as possible. As implied by Brandes (in Borich I974:88),

the most important determinants of the criteria, depth and quality of the

evaluation instrument are the reasons to evaluate.

The implication from the above is also that In order to accomplish the task of

evaluation, provincial education departments have to design a criteria-bound and

curriculum compliant evaluation instruments. In view of the narrow gap between

the grades, the GDE designed an inclusive Foundation Phase and Intermediate

Phase evaluation instrument. However, due to complexity in the subsequent

Phases and grades and also the implementation plan of the national Department

of Education, a grade specific instrument may have to be designed and

implemented per grade. This has been the case with instruments that were

designed for curricula prior to the RNCS (GDE 2003:4).

Public and private organizations, including the GDE, invest millions of rands in

programmes and projects. This implies that they have a remarkable

accountability to sponsors, stakeholders and other affected parties. Therefore,

for fiscal purposes, project leaders have to conduct and monitor an evaluation

process at certain points in the development of a project. As suggested by

Boulmetics and Dutwin (2000:10), if an evaluation report communicates results

that indicate success, it could assist the project leaders to obtain funding for the

current and forthcoming year(s). In that case we may assert that the mandate

for evaluation is for fiscal purposes or that receiving the second year’s funding

may be based on the results of the evaluation. It is also for when a GDE LTSM

Head presents evaluation results, which could assist the senior managers on

deciding on the worth of the programme based on its successes and failures.

19

As noted, above, the success of the evaluation process is determined by its

participatory nature. This is evident in the way the GDE models its evaluation

process. As implied above, educators at the Evaluation Centre are grouped in

teams of three. This implies that an individual evaluator will first evaluate a

LTSM submission on his/ her own but later on, the team consolidates its reports

into a composite document.

At school level, a teacher works collaboratively with fellow Learning Area/

Programme educators and later on they communicate their decisions to the

School Learning and Teaching Support Materials Committee (SLTSMC). The

SLTSMC facilitates LTSM management at school level which needs demands a

great deal of time (Monyokolo 1993: 16). The results at the Evaluation Centre

serve as the provincial GDE decision on what should be used in the classrooms.

The result at school level informs the educators as to what kind and which

materials the educators desire to use in their classrooms.

The above-stated approach is not only encouraged at Evaluation Centres but

also at other evaluation sessions that take place at the schools. These involve

the SLTSMC, Learning Area/ programme committees and grade educators.

More interesting is that evaluators have also expressed the difficulties they face

as they engage in the evaluation process irrespective of their working teams and

consulting with one another.

The above-named difficulties have been communicated to officials at Head office

and District offices. This is done by means of a variety of ways, including

telephone calls and direct personal contact with LTSM units. By citing the

problems the evaluators were sending a message that they required some kind

of support from the GDE as they felt this was lacking.

The foregoing scenario attests to the notion that another formal reason for

evaluating is to justify a project or programme. The project leader may have

brought the idea to the department or organization he is working for and will also

20

be conscious that he or she is always under the scrutiny of other administrators,

managers, parents, teachers and publishers. These stakeholders may only

desire to communicate support provided the evaluation results are backed by

statistics.

In support of the above, Boulmetics and Dutwin (2000: 11) argue that:

Gut feelings, perceptions, innuendo, and anecdotes are comforting but

they are not convincing to who require more objective evidence. Even

data are not always effective in convincing people, but data at least act as

a common currency to demonstrate the value of your case.’ (Boulmetics &

Dutwin 2000:11). Furthermore, one may expect a good manager to

collect data on a continuous basis in anticipation in order to justify the

need when the time comes.

The GDE could justify their evaluation process by the fact that educators as

evaluators of LTSM consider the context of the school community. Thus their

generic selection criteria would almost be the same for electronic as it is for

printed materials. This would reflect utilization of technologies to support the

processes (Eastern Cape 2003: 17).

Stufflebeam (in Kellaghan et al., 2003:31) presents Context evaluation as the

first step in his Context Input, Process and Product Context Input, Process and

Product (CIPP) model of evaluation. In this model, ‘Input’ applies to the amount

of academic and professional skills, knowledge and experience that the

educators/ evaluators are expected to put in the evaluation of LTSM. Through

their experience evaluators are able to make their informed contribution and

assist in making reliable assumptions and conclusions that are influenced by the

evaluation process.

Mezirow (2000:17) refers to the above-stated assumptions as habits of mind.

The habits of mind present themselves in a variety of ways: by sociolinguistic

21

(cultural or ideological) means, moral-ethically (conscience), epistemic (learning

styles), philosophically (religious or philosophical doctrines), psychologically (self

concept and personality traits), and aesthetically (values and attitudes). These

varieties also influence an evaluator prior to taking a decision whether the

materials they evaluate are appropriate and may be utilized in the classroom. In

support, McKinney (2005: 5) supports that by stating that textbooks act as an

interface between the officially state-adopted and sanctioned knowledge.

The GDE is presently focused on evaluation of printed materials. In future, the

process will change as the focus shifts to evaluation of electronic media. At the

time of writing this study, this was imminent, as the GDE had begun installing

computer hardware in its schools. Once the infrastructure has been installed the

next step would be the evaluation and selection of software. The product is

almost the same in the sense that both evaluations will end up with a Review

Guide (GDE LOETA policy 2003: 1).

The justification reason connects to the next one, which is comparison reason.

In that respect, for comparison reasons, the project participants consider how

their project compares with other similar projects that are running at present or in

the past. That is another way to justify the project. The project leaders have to

be objective and try to consider if there are any other projects they would have

done instead of the current. This implies that the project leaders would also

need to audit the project needs and consider whether they are justifiable as

viewed by clients, colleagues, sponsors and other stakeholders.

This also means having a checklist to ascertain whether the project has or will

achieve the intended outcomes set out at its inception. To be more specific, one

may explain an outcome as what happens as a direct result of an action – this

could include training, service, teaching, orientation, work shopping. Boulmetics

and Dutwin (2000:25) argue that an evaluation takes place to determine whether

an action was effective or not. More interesting an outcome is viewed not only

as what was anticipated but also what may come as a surprise (a surprise may

22

either be positive or negative in nature). Boulmetics and Dutwin (2000:25) refer

to these as anticipated and unanticipated outcomes. If an outcome is achieved

on a longer range it may be termed ‘impact’.

The factors discussed above suggest that evaluations are done for decision-

making purposes. Every organization needs to make plans and policies.

Evaluations assist managers in deciding whether an area under a particular

project is worth adopting as policy or not. Managers use evaluation results to

distinguish between best-working programmes and those that need to be

discarded. This inquiry may also assist planners of evaluation of LTSM in

deciding whether the project is sustainable. Furthermore, evaluation is

performed to expose all the participants to new opportunities and new solutions

to current problems.

2.4 CHOOSING A MODEL FOR EVALUATION

People involved with evaluation of programmes are faced with a large spectrum

of evaluation models which they may or may not adopt to finalize their evaluation

exercise. More often a combination of more than one model is used for a given

evaluation purpose. In this section, I will discuss some of the popular models of

evaluation as suggested by Boulmetics and Dutwin (2000) and backed by the

views and contributions of other professionals.

2.4.1 Discrepancy Evaluation Model

Developed by Provus (in Boulmetics and Dutwin 2000:5), the Discrepancy Model

is based on the premise that a project is not in a vacuum but falls within an

organizational structure. The aim is not to determine the cause-and effect

relationship but to understand the evidence that is found based on an evaluation

exercise. The emphasis is on why events took place and not on whether or not

they occurred.

23

This model’s development takes place through stages of design, installation,

process, product, and cost-benefit analysis. There is a set of standards put to

examine performance. This is in line with what Heinich, Molenda, Russel, &

Smaldino (1996:37) regard as evaluation being governed by pre-assessment

measures. These pre-assessment measures should be communicated with all

the evaluation participants prior to the evaluation activity.

Ideally this evaluation type should therefore start with a meeting where an

evaluator’s programme description is worked out. Masokoane (1993:64 in

Kromberg, Govender, Birrell & Sibanyoni) refers to this consultative process as a

‘moral and ethical framework of good practice’ and ‘democracy, quality and

transparency’. This description has to match the standards for evaluation. To

measure success, a comparison is made between what is being accomplished

and what was anticipated. This comparison precedes a cost-analysis process in

which the project costs are compared with the cost of similar projects.

2.4.2 Goal-Free Model

As stated by Propham (1974 in Boulmetics and Dutwin 2000:73), the Goal-Free

Model was developed by Scriven who during evaluation looked at a

programme’s actual effect on identified needs. Evaluation is done on what the

project is doing to address the needs of a client or client population. In this

model observation is conducted without the use of a checklist. However, data

is accurately recorded to determine the significance of the findings. At the end

of the evaluation process, the evaluator will have a descriptive analysis of the

results and involve others to participate in such analysis.

The main characteristic of this model is that the evaluator does not commence

the evaluation process with any preconceived notions of the outcome. Scriven

(in Borich 1974:30) describes goal-free evaluation (GFE) as the one in which the

evaluator performs the function from a more general perspective. This also

implies that the organization is not supposed to make any goal statement as this

24

might derail the evaluator and obstruct him/her from performing the evaluation

successfully. In this model, the evaluator may employ both obtrusive and

unobtrusive methods to gather data. The former refers to using methods or

equipment that the subjects are aware of (for instance tests), while the latter

refers to those that the subjects are not aware of (for instance hidden tape

recorder).

Based on the above, Rosi and Freeman (in Boulmetics & Dutwin 2000:74),

state that people’s views on the use of the model are different. Some think that

the evaluator has to be an expert in the field to be evaluated whilst others think

such is not necessary. In evaluating LTSM, it is important that the evaluator

possesses skills in evaluation and knowledge of the Learning Area for which

materials are to be evaluated. The lack of the skills and knowledge might

jeopardize the integrity of the evaluation exercise.

2.4.3 Transaction Model

The Transaction Model was developed by Stake, (Boulmetics & Dutwin 2000:75)

concentrating on activity between the evaluator, participant and the project staff

(participants). It is aimed at benefiting the clients and practitioners. In this model

there is a continuous two-way communication (feedback) between the evaluator

and organizational staff. The approach is participative in nature and the

evaluators and clients are viewed as equal partners. The evaluator uses a

variety of interview and observational techniques to gather data. The orientation

can have a goal-free or goal-based technique.

2.4.4 Decision-Making Model

25

This model, developed by Daniel Stufflebeam (Boulmetics & Dutwin 2000:75),

is used by leaders to predict the future of the project, which in turn affects the

future of the organization. The concern is about the long range effect of the

project on the organization or the client population. It is not about the present

situation. It assists leaders in making decisions in the future, such as cutting

the costs. Most importantly, the relevant stakeholders are invited to partake in

the decision-making processes.

2.4.5. Goal-Based Model

Due to the fact that it is easy to use, the Goal-based Model is the most

commonly used. Unlike in the goal-free model, in this model the organization is

free to set and present goals or objectives to the project leaders. The goals are

set according to the needs of the organization and plotted on either a proposal,

brochure or project description. Furthermore, Boulmetics & Dutwin (2000:77)

emphasize that the wording of the objectives would usually set and identify the

standards for evaluation. The evaluator uses either qualitative or quantitative

methods to measure the outcomes, objectives or goals that were set by the

client or organization.

Although the above models are interdependent, this research project adopts

mainly the transaction model. This is what the GDE may be aiming at and often

there is mention of social aspects like collaboration and participative ness.

2.5 CRITERIA FOR EVALUATING LEARNING AND TEACHING SUPPORTMATERIALS

Criteria for evaluating LTSM form the basis for the instrument used for evaluation

purposes. However, various authors and experts on evaluation give various

arguments on what needs to be part of the instrument for LTSM evaluation. This

section of the research explores some of those suggested criteria. Most of the

suggested criteria are generic in nature as compared to what is contained in the

26

instrument. For this exercise, the suggested criteria will be juxtaposed with

criteria that are contained in the GDE evaluation instrument (GDE LOETA policy

2003:1).

Heinich, Molenda, Russel & Smaldino (1996:36) state that an instrument for

evaluation should allow for a superficial analysis of learner’s characteristics. This

implies that there should be consideration of the learner’s level of development,

age, cultural background and values. Furthermore, there is an argument that if

an instrument makes such consideration this could provide helpful leads in

selecting LTSM.

In the GDE instrument, this coverage extends to almost all the sections. For

instance, under compliance with the curriculum, there is a criterion that deals

with establishing whether the materials are in line with the constitution of the

country in terms of sensitivity to issues of gender, age disability and HIV/ AIDS.

Previously, with some exceptions, the general trend in textbooks and readers,

was over-representation of males on covers, main characters, other characters

and images inside a book (McKinney 2005:13).

Furthermore, McKinney (2005:19) stipulates that most books show a severe

under-representation of black characters and images of disabled persons. The

issue of gender characteristics, particularly masculine and feminine stereotypes

remains an area of concern and needs to be investigated by means of LTSM

evaluation instruments (Evans & Davies 2000: 5).

The reason why the GDE instrument stresses this criterion is probably to avoid

the reoccurrences of the past discriminations. The evaluator is also requested to

consider whether the LTSM cater for learners with varied abilities as stipulated in

the Inclusion/ Education White Paper (GDE 2003: 7).

Furthermore, Heinich et al. (1996:37) argue that in case of a heterogeneous

group, evaluators should be assisted to select LTSM that provide for a common

27

experiential base. This is in line with what the GDE instrument illustrates as

context, especially the criterion that seeks to ascertain that learners are given the

opportunity to discover the world that is beyond their own. This connects with

recognition of various social structures; balance between the learners’ familiar

and new information; and acknowledgement of the learners’ context and

experience.

The above mentioned criteria regarding the level of development also include the

language level. This is evident in the argument by Potenza, (in Kromberg,

Govender, Birrell & Sibanyoni 1993:50) that sensitivity to language should be

included be considered when evaluating LTSM. However, oversimplification of

language like English for second language users should be avoided.

In addition to the above criteria, Heinich et al. (1996:45) state that the LTSM

should match the intended outcomes or objectives. At this stage, one is

reminded that the intention is to obtain the appropriate LTSM by selecting from

available LTSM, modifying existing or ‘off the shelf’ LTSM or designing new

LTSM. Irrespective of the category, the LTSM must meet the required standards

in order to be selected.

The above-mentioned criteria would be incomplete without the following checklist

as suggested by Heinich et al. (1996:47):

• Does it match the curriculum?

• Is it accurate and current?

• Does it contain clear and concise language?

• Will it arouse motivation?

• Does it provide for learner participation?

• Is it of good technical quality?

• Is there evidence of its effectiveness (e.g. field-test results)?

• Is it free from objectionable bias and advertising?

• Is a user guide or other documentation included?

28

After having gone through the evaluation criteria, one needs to learn that their

application is a specialized process. As implied by Heinich et al. (1996:46),

selection criteria vary with different media formats. In other words, criteria that

apply in one type of LTSM may not necessarily apply in another. For instance,

criteria for selecting a chart may not be the same and not even as detailed as

criteria for evaluating a book or atlas for that matter. To address this need, the

GDE evaluation instrument consists of different sections, each stating the type of

LTSM it may be used for, thus the following:

• Section A: To evaluate Learners’ Books, Learners’ Workbooks/ Activity

Books, Readers, Educators’ Guides and Books

• Section B: To evaluate Learners’ Books, Learners’ Workbooks/ Activity

Books, Readers, Educators’ Guides and Books

• Section C: To evaluate Games, Educational Toys, Models, Puzzles and

Audiovisual Materials

• Section D: To evaluate Charts and Posters

On the other hand, criteria for evaluation may apply to selected, specific learning

programmes, for example the GDE instrument is learning programme specific.

For instance Section B is designed to specifically evaluate LTSM for Literacy,

Life Skills or Numeracy (whichever is applicable).

Some of the criteria suggested by Heinich et al. (1996:46), consider objectives,

audience, cost, technical expertise, equipment and time. It is also suggested

that a preview or screening of LTSM happens before the actual evaluation

process. This could be done by using techniques such as reviews, blurbs or

appraisals by colleagues. This implies that evaluations are done before, during

and after utilization (Potenza in Kromberg et al., 1993:48).

.

29

If an education department is able to apply its evaluation criteria, techniques and

skills accordingly, it will be able to eliminate practices such the ones that Potenza

(in Kromberg et al., 1993:48) is objecting to i.e. publishers reprint new editions

of old textbooks instead of publishing new materials. This act could be ascribed

to corruption in the publishing world particularly because some publishers

knowingly submit materials that contravene the laid down standards and criteria

for selection. They often hope to use their unacceptable influences to ensure

that their inappropriate LTSM are selected (Proctor and Monteith in Kromberg et

al., 1993:33).

2.6. SUMMARY

In this section, using different sources, I explored different definitions of the

process of evaluation and constructed my own definition of the concept. I further

tried to contextualize the concept to the evaluation of Learning and Teaching

Support Materials using an instrument designed by the GDE. I also illustrated

that evaluation happens for a specific reason that will benefit the organization or

client population in one way or the other. Some of the prominent reasons for

evaluation are for the organization to increase efficiency, to make sure that

projects are effectively implemented and to increase impact of activities involved

in the project.

In order to achieve the above-named aims of the project, project leaders have to

understand different models and adopt one for their project. However, my

argument is that these models are not exclusive in nature but can be integrated.

The discussion would have been incomplete without an examination of criteria

for evaluating LTSM. To attain that, I explored, compared and contrasted criteria

as suggested by different authors and against the instrument that is used by

GDE to evaluate and select LTSM.

Furthermore, I explored different models for evaluation of LTSM. The implication

is that when one evaluates an LTSM evaluation project, components of each

30

model should be considered. A lesson is learnt from Provus’ Discrepancy model

in which there is emphasis on always remembering that a project is not in a

vacuum but part of the organization. Therefore, the GDE LTSM team should

ensure that their project is within certain specific organizational parameters.

This is also in consideration of the fact that a project should be managed in a

participative manner.

Furthermore, a project would be worthless if the needs of the client are not

considered. This is expressesed in Propham’s Goal-free model. The two-way

communication is supported by Stake’s Transaction model in which the

practitioners’ significance is highlighted. This presents all the participants with an

interactive, communicative and collegial relationship.

In this section, there has also been an emphasis on the long-range effect of the

project. Even when the project work has come to an end, the clients or target

population group should continue to enjoy the benefits of the project. This is

expressed in Stufflebeam’s Decision-making model. It is supported by the Goal-

Based model that stresses that the goals should be specific and also meet the

needs of the population.

The next section focuses on the research methodology employed in the study. It

describes the research design and explains the data collection tools which will be

used.

SECTION THREE

RESEARCH METHODOLOGY

31

In this section, I will firstly discuss the characteristics of qualitative research in

more detail. Furthermore, qualitative and quantitative research is juxtaposed.

Data collection methods that will be used are also introduced in this section.

Later on in the section, the research design is explained in conjunction with the

population and sample that was used to gather data. There is also an exposition

of the way in which the focus group interviews are conducted. The intention is to

utilize focus groups to promote discourse among the participants. Discourse is

the way to achieve the intended outcome of focus groups which is to share

experience around areas of practice (Wiesener and Mezirow, in Mezirow 2000:

332). This is coupled with the list of questions in the interview schedule. Since a

survey questionnaire is used, it is also vital to discuss its validity and reliability.

3.1. QUALITATIVE RESEARCH

The qualitative approach adopted in this study is characterised by the human

mind that plays the role of being an instrument of research and that is referred to

as positivist discourse of social enquiry (Henning et al.,, Van Rensburg and Smit,

2004:7). This implies that the researcher assumes an Interpretivist role (Trauth

in Henning et al., 2004:9) and that is instrumental in ensuring that the possible

error associated with observation is avoided. The findings of the study are

backed by Literature reviews which evaluate and analyse materials that have

dealt with related subjects in the past.

The aforementioned implies that as opposed to quantitative research, in this

study the participants are free to communicate other information that is not

explicitly expressed on the artefacts (Henning, et al., 2004:3). The variables are

not controlled and the participants are free to respond and naturally develop as

the research work progresses. This is made possible by means of focus group

interviews and observations. Through these data collection methods, the

research is able to delve into depth of the variables.

32

The data is gathered by participatory observation, artefact (using survey

questionnaires to evaluate the GDE’s Evaluation instrument) and document

studies and in-depth interviews. Design types include discourse analysis,

interviews and surveys. The design methods are categorised as Ethnographic

studies, Discourse analysis and Qualitative evaluation tool. The researcher is

required to ensure that there is validity, reliability and generalisability in the

instruments and processes that take place (Henning, et al., 2004).

3.2 RESEARCH DESIGN

Having recognized and concluded that teachers who evaluate materials need to

be provided with a useful, valid and reliable evaluation instrument, a research

outline was developed to investigate the matter. The inquiry was initiated with

Foundation Phase teachers who were expected to have interacted with the

evaluation instrument as individuals or groups at different levels including school

level and at the exhibition centre. The logic of the data was that it would make

clear how useful the evaluators find the artefact, i.e. the evaluation instrument.

Since it is expected of schools and teachers in different areas of Gauteng to

have used the evaluation instrument, I decided to gather the data from a

heterogeneous group of 157 potential LTSM evaluators at a training centre

organized by the Ekurhuleni West District of the GDE. This implied that the

inquiry was done on educators from schools that are rated differently as per the

poverty index and were also from different cultural groups.

The selection criteria for evaluator participants were within the following

parameters: teachers currently teaching in the Foundation Phase and who are

involved in teaching based on the Revised National Curriculum Statements

(RNCS). The search for the educators who could be part of the research started

with an interaction with my colleagues in the GDE’s Teacher Development unit.

This interaction involved arranging to have a time slot during the provincial

training of educators on the RNCS to hand out the survey questionnaire and

33

explain what was expected of the teachers (e.g. the return of completed

questionnaire) (refer to Addendum D)

To arrange for the above stated, a formal request had to be submitted to the

Senior Manager of Human Resources Development for approval with the aid of

the attached letter to request permission. Conditions for the data collection

process to take place with minimal or no interference with the training process

were negotiated with the teacher development unit personnel. These

negotiations were made possible by accessibility of the GDE Project

Management Team.

Since the permission was granted I went to the training centre and distributed the

artefacts and requested that the training coordinators collect at the end of the

session. This assisted in ensuring that all the completed instruments were

returned. The teachers did not take the instruments home and that made it

possible for a hundred percent collection. Furthermore, I selected a group of

ten participants for a focus interview and also observed them as they trialed the

evaluation instrument against samples of LTSM.

3.3 RESEARCH ORIENTATION AND CONTEXT

The research survey was conducted at the Elspark Secondary School which was

used by the Ekurhuleni East District as a training centre for educators on RNCS

(including how to evaluate LTSM). The educators were requested to complete

and re-submit a survey questionnaire (See Addendum E) on the usefulness of

the LTSM evaluation instrument. Arrangements were made for focus interviews

to be held at a later stage at the different school locations and clusters. The

conditions under which the focus group interviews took place were strictly private

in a room at a school. Interview schedules were designed and used as guidance

during the interviews. This was made possible by the GDE Head Office Teacher

Development unit, the training Project Management Team, Training co-

34

coordinators at the training centre and Higher Education Institution

representatives.

3.4 SAMPLING

Although the study is predominantly qualitative in nature, the type of questions

asked in the survey questionnaire is a combination of qualitative and quantitative

inquiry. This questionnaire is administered on the participants who were

identified and formed a sample. These participants were selected by means of a

combination of purposive and theoretical sampling as people who fit the criteria

of desirable participants were selected. These 157 trainees of Ekurhuleni East

District were selected on those premises as they probably possessed the

potential of actively participating in the process in a social interactive way (Refer

to Addendum B).

Furthermore, a smaller group of ten participants was selected to participate in

the focus group interview. During this interview the evaluators were engaged in

a reciprocal social interaction in which they and the researcher shared their pre-

existing knowledge and experience in order to socially construct knowledge on

issues that are pertinent to the evaluation of LTSM in the GDE. It is through this

discourse that the participants learnt that they were equal partners and were also

free to make their contribution in this dynamic social context in which

construction of knowledge took place (Gravett 2001:20).

3.5 DATA COLLECTION METHODS

The primary data collection method was the survey questionnaire and the

secondary data gathering method was the focus group interviews with ten

participants that were randomly selected. Participatory observations were done

as the participants were given the instrument and samples of Foundation Phase

books to evaluate. Participatory observation was used as an experiential frame

during the interpretation of data. This was coupled with what Henning et al.

35

(2004:7) term document analysis. A brief discussion of each of these methods

will follow in the passage below.

3.5.1 Survey Questionnaire

As stated above, the primary data collection method is a survey questionnaire

and it was administered prior to the focus group interviews and observations.

Considering the vast nature of the LTSM process as depicted on the process

map above, caution had to be exercised in the designing of the artefact, the

survey questionnaire. The items of focus were first determined prior to the

setting and classification of questions (Henning et al.,, Van Rensburg and Smit

2004:89). This has prompted the classification of the questions into sections,

namely Personal Information, Evaluation Experience and Actual Engagement

with the Instrument.

What is important to note is that the instructions to the participant are on the

initial part of the questionnaire (See Annexure A). In constructing these

instructions, I considered that there was unequal power between myself, the

researcher and the evaluators as participants. Because of my position in the

Department of Education. The evaluators probably regarded me as an authority

and may have been threatened and tense. The instructions were a way of

relaxing them, particularly as they were alerted that they were going to be treated

as partners and would also remain anonymous. The primary aim was to make

process remain a social interaction. That would also ensure that there is true

knowledge sharing (Henning, et al., 2004:67).

3.5.2 Focus Group Interviews

36

As stipulated above, the aim of this inquiry is to put together evaluators who have

the necessary pre-existing knowledge and experience to partake in a social

interaction in which new meaning and knowledge is constructed on issues

involving LTSM evaluation and selection. The focus interview therefore creates

a context in which there is reflec tive discussion in which the various participants

bring a range of opinions and experiences as outlined by Morgan (in Fortuin

2002:53).

The questions I asked will be discussed below:

3.5.2.1 Interview schedule

The open ended questions that were asked were as follows:

• Tell me about your understanding of the process of evaluation of the

salient features or steps in the LTSM process map.

• What are the challenges that face the GDE and its institutions in ensuring

that appropriate LTSM are selected?

• If you were given the opportunity what amendments would you introduce

in the LTSM evaluation process?

• Do you regard the involvement of educators in the evaluation of LTSM as

a vital step that was taken by the GDE?

• Tell me about an experience that you will never forget in your involvement

in LTSM evaluation.

• How does the LTSM evaluation instrument offer an educator support in

selecting appropriate LTSM?

3.6 VALIDITY AND RELIABILITY

3.6.1 Validity

37

As asserted by Silverman in Fortuin (2002:61), this section is about validity which

is concerned with the accuracy of findings in terms of the reality it reflects. The

GDE evaluation instrument was measured to ensure that it complies with validity

requirements. The validity of this qualitative study is established by the detailed

audit trail which includes evidence of data collection analysis and findings. The

findings as discussed in section three above may be classified as primary data

that was derived from the survey and focus group interviews. The evaluators’

experiences were captured by means of the various data-collection techniques

which were set to possess the validity requirements.

Considering the contribution made by Silverman in Fortuin (2002:61) that

interviews alone can become unreliable at the level of method, I also considered

to precede their utilization with the use of a survey questionnaire which also has

to be a valid data collection instrument. Among other things, I endeavored to

use clear, concise language in both the data collection techniques. The

questions used in the focus group interviews were refined to be as friendly

enough to the interviews. I was quite flexible with the language of

communication as the participants were competent in different languages

although all are comfortable users of the English language. I also applied code-

switching when I gave them some instructions or unpacked the contents of the

survey questionnaire. The transcription of the focus interviews was checked by

my English-speaking colleagues.

3.6.2 Reliability

Reliability may be interpreted as the consistency between the findings and the

methods of data collection and analysis (Merriam, in Fortuin 2002:62). The data

collection methods that were used are appropriate to the research question. The

current study therefore should have some relation with subsequent enquiries in

the same field. All processed data has been made available for further perusal.

38

As suggested above, reliability is applied in order to ensure that there is

consistency with the data from focus groups. This assisted in revealing the

experiences of the evaluators of LTSM even when the survey was repeated the

same results were achieved. This also applies to the first and second sets of

focus interviews.

3.7 ETHICS

This study complies with the ethics of the University of Johannesburg as

stipulated in its Academic Ethics Committee Document that was used for

discussion on 5 April 2005. When the survey questionnaire was handed to the

participants, they were told that their participation was voluntary. This was

important because most of them knew the researcher and could have felt obliged

to participate because of his position in the GDE. In addition, the participants

were told about the purpose of the survey: that it was not part of the training

session or a duty to be compulsorily done. Their confidentiality and anonymity

was assured on the first page of the survey questionnaire. All these principles

were adopted for the participants to accept and have trust in, particularly by

ensuring that they do not suspect that they are being betrayed. Furthermore, as

stated in the ethics guidelines of the University of Johannesburg (2005: 1) it was

indicated to the participants that permission to conduct the study was granted by

the Teacher Development Unit.

3.8 SUMMARY

This section was focused on data collection and analysis. The focus of the study

is determined and declared by means of research orientation. The context of the

study is analyzed and stipulated to show the extent of the research and the size

of the population and sample. This is coupled with the explanation of data

collection methods applied, surveys and focus group interviews (using interview

schedules). The concepts of validity and reliability are also discussed.

Furthermore, the interview questionnaire and other instruments are tested for

39

effectiveness and efficiency. In the next section data are analyzed and the

research findings emerge.

SECTION FOUR

DATA ANALYSIS AND RESEARCH FINDINGS

40

As indicated in the previous section, this inquiry sought evidence for the

usefulness of the instrument used by educators as evaluators of LTSM in the

GDE schools. The previous section focused on the research design and

research methods, namely the research survey and the focus group interviews

and their related tools (survey questionnaires and interview schedules) (Henning

et al., 2003: 7).

In this section, however, research findings will be presented based on the

research literature that was explored in earlier sections. The analysis of data will

be backed by, among others, the work of Henning et al. (2004) in which the

researcher is advised and given skills how to presented data in a narrative way.

There is also an adaptation of the constant comparative method as suggested by

Maykut and Morehouse (1994) in which data is grouped into sub-themes and

themes called codes and categories respectively. Furthermore, there is

implementation of the constructivist approach as viewed by, among others Scott

in Gravette (2001).

4.1 ANALYSIS OF DATA

The findings of this inquiry are a result of the steps undergone in the process.

This process began with a survey that was made possible by means of a survey

questionnaire. The questionnaire is used to develop an understanding of the

challenges experienced by educators as evaluators of LTSM when engaging with

the instrument used to evaluate LTSM. This was followed by focus group

interviews in which ten participants were interviewed for a further data gathering

process. As suggested by Henning et al. (2004: 101), in this section. I shall in

this section make a narrative presentation or analysis of my understanding of the

data that was collected by means of the aforementioned methods.

Interestingly, there is an element of interrelationship between the methods of

data collection, as these themes are covered by the various sections of the

survey questionnaire.

41

As indicated below and as suggested by Maykut and Morehouse (1994: 127),

based on the focus group interviews, the following themes were identified. Each

theme develops from codes that were constructed by grouping the responses

given by users of the instrument to evaluate LTSM (Refer Table 1). As the

discussion progresses, it is also important to note how the codes overlap and

thus a code may develop to more than one theme:

• Willingness among educators to perform the function as part of theirwork

This theme has some relationship with and develops from the codes identified

as:

Inadequate support to institutions, Instrument used by different sections in the

GDE, Instrument well marketed within the GDE, Effective training sessions,

Instrument addresses the needs (including curriculum), Instrument enables the

educator to decide on LTSM, Instrument used in collaboration, Instrument boosts

confidence in evaluation and Evaluation is indispensable.

• Evaluation happens with some assistance

The corresponding codes for this theme are: Inadequate support to institutions,

Effective training sessions, Instrument used in collaboration, Instrument boosts

confidence in evaluation, and Evaluation is indispensable.

• Instrument is the core element for evaluation process

This theme relates with codes like: Instrument used by different sections in the

GDE, Instrument well marketed within the GDE, Effective training sessions,

Instrument addresses the needs (including curriculum), Instrument enables the

educator to decide on LTSM and Instrument boosts confidence in evaluation.

42

• Instrument encourages collaborative evaluation

This section makes provision for statements like: Instrument is used by different

sections in the GDE; Instrument is well marketed within the GDE; Instrument

contributes to effective training sessions; Instrument addresses the needs

(including curriculum); Instrument enables the educator to decide on LTSM;

Instrument is used in collaborative processes; and Instrument boosts confidence

in evaluation, and Evaluation is indispensable.

As illustrated by Maykut and Morehouse (1994; 127), the constant comparative

method is a useful method to analyse data. In this form of analysis data is

interpreted and categorised for the purpose of illuminating and elucidating the

research problem that would have been stated by the researcher. The data is

further sorted and named, with a constant comparison of new codes and

categories (Fortuin (2002; 56)

In relation to this study, the above codes were identified and analysed

considering the social constructivism basis of this argument. The manner in

which the evaluators of materials engage with the instrument and the types of

responses that form codes and categories in the above table, are evaluated

against the premises of social constructivism. The evaluators’ prior knowledge

of evaluation is accepted and utilized as a useful aspect of the evaluation

process. To make possible a social interaction, the project leaders do not use an

individualistic approach in which evaluators are required to work as individuals.

Instead, the evaluators are put in a room in which they are able to interact with

one another. During that interaction, the evaluators learn from one another and

together create meaning and knowledge of what is expected of them as

evaluators. This also applies to the way in which they put together reports on the

LTSM that they are evaluating.

When the focus group interviews take place, the aforementioned atmosphere

that promotes interaction makes it easy for the evaluator to codify the

43

participants’ responses to some of the questions which are based on the survey

questionnaire. The researcher makes the process as participative as possible by

relaxing the evaluators so that they are not conscious that he is recording the

interviews. At the end of each session, their opinions are also considered.

In my attempts to explain, I had to systematically proceed from the organizing

and coding of data, grouping of data into categories, to developing rules in order

to include all the units in each category and lastly reflect on the responses to the

research question (Fortuin 2002:56).

Reading through the focus interview transcriptions (see Addendum F) enabled

me to have a better understanding of the frustrations and problems experienced

by evaluators of materials in their engagement with the evaluation instrument.

Their responses included expressions like:

• “need more support”

• “better when working with others”

• “feel more secure”

• “managers hide away information”

• “learn a lot from Evaluation Centre”

• “need to do it more often”

• “once per annum is not enough”

• “no sense when education department does it for us”

• “feel owning the process”

• “educators should receive even more incentives”

• “should be part of our careers”

These responses were classified and categorised into the codes indicated in the

above discussion.

4.2. DISCUSSION OF THE FINDINGS

4.2.1 Motivation for educators to be evaluators

44

It is interesting to note that during the focus group interviews, some evaluators

expressed a wish for the evaluation function to form part of their work and be

included in their job descriptions. However there seemed to be mixed feelings

on this issue. For instance, Simon1, a respondent stated that: “remember aseducators we receive meagre funds… I am therefore not willing to work forno pay” (see Addendum F).

The above statement implies that some educators view themselves as a group

distinct from their employer. This also implies that among educators there is still

a “we-they” polarity as portrayed by Burtman and Borich (Borich 1974:39).

However, such educators do not consider the fact that evaluation is a means to

empower them in the name of transparency and democracy as suggested by

Masokane (1993:64 in Kromberg, et al.). It can also be concluded that some

educators take the formal evaluation process as an opportunity to raise some

income. This notion is supported by the view of Borich (1974:27) that evaluation,

like the cost of living, has as its aim a propensity for rapid change.

Nevertheless, I argue that we may not conclude from this statement that among

the educators there is total unwillingness to partake in the evaluation process.

This is because the respondent does not complain about being part of the

process. Evidence for that willingness is strengthened by Sandile’s remark when

he said: “remember that it is your calling. Besides, it is within the principlesof democracy that we should have a contribution to make in ourprofession” (see Addendum F). This statement also implies that some

educators would like to have the evaluation process as a non-optional

professional demand by the GDE on its educators (Kellaghan, et al., 2003:17).

Besides for the above-stated reasons, some educators are eager to have

appropriate materials in their classrooms. They think by forming part of the

evaluation teams they will own the process and will, as Mary said: “…selectmaterials that do not have any weaknesses that could hamper or negatively

1 In this study I will use pseudonyms in order to preserve the anonymity of the respondents

45

affect the teaching and learning processes in our schools” (see AddendumF). Besides, this involvement will also entitle the educators to a role of

partnership with education officials.

Furthermore, educators as participatory evaluators of materials claim that even if

they may engage in voluntary evaluation or perform that as part of their work,

they will still need the backing of senior officials in the department. Sandile

voiced that when he said: “… we will feel more secure and protected if wehave the backing and support of our line functionaries in particular if ourprincipals could refrain from blocking dissemination of information” (seeAddendum F). In other words this statement shows a concern on outcomes

which reveal the impact of their involvement and participation in the process of

evaluation. This is in line with the view that impact refers to the attainment and

non-attainment of outcomes (Borich 1974: 88).

As implied earlier on evaluators are engaged in different kinds of discourse.

First, it is their discourse with the text or titles that they evaluate. Second, it is

the discourse with fellow evaluators of materials. The internal evaluators are

willing to have such a discourse on a daily basis. At school level, educators who

are evaluators can enter into contrastive discourse which could provide socially

relevant findings and insights which could be instrumental in their selection of the

LTSM on behalf of the school (House 2000:598).

As suggested in section two above, the aforementioned still implies that

educators are involved in a dialogic construction of knowledge with their fellow

colleagues (Gravett 2001:20). This is simply because the instrument

encourages an educator to engage with others in the process of evaluating LSM.

This kind of discourse manifests itself in the form of discussions.

The notion that other respondents argue that the evaluation process is a

professional demand by the GDE on its educators, was evident when Maxwell

said: “At times I think they are abdicating their responsibilities because

46

they leave us without support at all” and in support, Sandile stated: “it iswithin the principles of democracy that we should have a contribution tomake in our profession. We need to be proud by knowing that we werepart of the selection process” (see Addendum F). The general feeling is that

the evaluators are bound by the predetermined standards decided by the

curriculum and reflected on the instrument for evaluation. This also correlates

with what was stated earlier that for effectiveness, efficiency and impact, the

process should not be goal - free as per the description given by Borich

(1974:30). This finding links with the following one as the evaluators feel that

while they are willing to perform the function they need some support in the form

of collaborative interaction.

4.2.2 Supported evaluation

According to the social constructivist viewpoint adult learners should be given an

opportunity to construct new knowledge utilizing their pre-existing knowledge and

experience by recognizing and reconstructing experiences encountered on the

social plane (Driver-Scott in Gravett 2001:20). Further, they use these

experiences to enrich and revise their knowledge. The need for this kind of

support is expressed by the evaluators of LTSM. They expect the GDE to create

such a social plane that comes in the form of a partnership between the LTSM

coordinators and themselves. This is evident from Bafana’s response when he

said:

Based on my experience as an educator, member of the materialscommittee and evaluator of materials, I would think that the GDE ismoving towards the right direction. In the past we were excludedfrom the process of taking decisions as to what kind of materials wewished to utilize to develop our children. Materials that we didn’tdesire were ordered on our behalf. We were not given theopportunity to accept or reject (see Addendum F)

47

This statement remains constructive for as long as evaluators do not enter the

process with preconceptions (Scriven in Borich 1974:30).

The evaluators do acknowledge the fact that their pre-existing knowledge and

experience are pivotal in the process of evaluating LTSM. However, even

experienced evaluators are not confident enough without some kind of support

that they expect the GDE officials to render. This is evident from Jeremy’s

response that: “It is actually a very complicated process when you are notgiven support”. Furthermore, Maxwell said: “Especially the so called districtofficials. At times I think they are abdicating their responsibilities becausethey leave us without support at all” (see Addendum F). I could argue that

this statement indicates a certain amount of insecurity on the part of some

evaluators. Jeremy’s point is actually refuted by the view that evaluators should

make their contributions that are influenced by their engagement in the

evaluation process as stated by Stufflebeam (in Kellaghan et al., 2003:33).

In fact, during the focus group interview interaction most of the interviewees

expressed a wish to partake in evaluation that takes place at the provincial level

rather than at school level where they believe there is lesser support. This is

clear from Hlomu’s response when he said: “They advise us to do that even atschool level. In this case you need to consult with other teachers via theLTSM committee to decide on the kind of materials to be selected for yourclassroom” (see Addendum F). However this is refuted by one of the premises

of the discrepancy model by Malcolm Provus (in Boulmetics and Dutwin 2000: 6)

which states that a project is not in a vacuum but falls within an organizational

structure.

From the above-stated responses one may conclude that there is a general

feeling that District offices do not render the expected support regarding the

evaluation of LTSM. However, the evaluators are conscious about the fact that

they need to have an input in form of academic and professional skills,

knowledge and experience (Stufflebean in Kellaghan et al., 2003:31).

48

The efforts that the evaluators put into the evaluation process would be fruitless

if they did not have reliable and useful artefacts. This forms part of the next

theme of the findings of this inquiry.

4.2.3 The Instrument forms the core element of the evaluation process

During the focus group interview evaluators who were involved in the evaluation

of LTSM prior to the introduction of the GDE evaluation instrument regarded the

instrument as a useful artefact. George, who had some experience, supported

that when he was asked whether he thought the instrument was useful, he said:

Oh! That instrument is my saviour. You know, I have always hadpassion for the exercise of evaluating the materials. My biggestfrustration was doing that without any instrument. I remember wewould sit together as a team and create our own criteria – some ofthose were not measuring what we intended to find out about thebook (see Addendum F).

George, in his statement, could be referring to the effectiveness which is in line

with what was cited earlier from the work of Boulmetics and Dutwin (2000: 25)

which states that an evaluation takes place to determine whether an action was

effective or not. Furthermore, this is an exposition of the need to have pre-

assessment measures that govern evaluation as suggested by Heinrich, et al.

(1996:38).

Absence of an instrument is probably as bad as having one that is subjective.

Jenny’s statement supports George’s earlier assertion when she said: “Iremember that took away a lot of trust. We were in the schools’ textbookcommittee and some staff members felt that we were favouring certaintitles at the expense of others” (see Addendum F). This is also in line with

the notion that in any project there is a set of standards stipulated to examine

49

performance (Heinrich, et al., 1996:37). The tendency was for them to express

the frustrations that they experienced in their endeavours to evaluate and select

LTSM for their classrooms. Some evaluators, as stated in the previous section

felt that without an evaluation instrument, the GDE would be denying them some

support.

Considering the qualities of the instrument, responses on the survey

questionnaire indicate some positive attitude towards the evaluation instrument.

This is also expressed on the group findings where 46% of the respondents feel

that a specific section on the instrument should not be changed they are happy

with the organization of the instrument.

The general aspects of the instrument include aspects like the LTSM’s physical

aspects (durability, safety, binding, and texture), colourfulness, font size, white

space, and objectivity. The specific aspects of the instrument test for the

technical, professional and educational qualities (including compliance with the

curriculum, relevance to the Learning Area, relevance to the assessment

requirements and the level of the learners’ development (GDE 2003:7-15).

Objectivity may further be ensured by avoiding the publishers’ knowledge of who

the evaluators are prior to finalization of the evaluation process (Denning 1992:

9).

The evaluators, particularly the less experienced, seem to be happy with both

aspects. However, the more experienced evaluators, some of whom received

formal evaluation training at tertiary level, have expressed some gratitude for the

specific, technical aspects of the instrument. They have even expressed that

this aspect was not explicit enough in the other kinds of instruments or

instructions that they were required to utilize before.

On the survey questionnaire, the 157 participants were asked whether they were

familiar with the GDE instrument. A total of 88 participants (56%) stated that

they were not familiar with the instrument at all while 69 (44%) were familiar with

50

it. The participants’ responses were grouped using the tally system, converted to

frequency. Those that were familiar with the instrument used it at varying

frequencies per annum, either once, twice, quarterly or monthly. These varying

frequencies are caused by a number of reasons some of which could be

obtained during focus group interviews. One may conclude that they find the

instrument to be an important element in the evaluation of LTSM.

When the educators were asked if the instrument needed to be improved or

changed, the majority thought that it needed to be adjusted. This is supported by

the findings as discussed in the next paragraph. Fikile, one of the respondents

expressed it more specifically when she said: “Our education system isdynamic and therefore whatever instrument is used needs to be adjusted tokeep up with the times”. Such adjustments are pivotal in ensuring that the

instrument keeps up with the needs of the client or client population as

suggested by Propham (in Boulmetics and Dutwin 2000:73) in his Goal-Free

model.

The findings on this issue were grouped and summarised as follows: of the 157

respondents, eight that the information on the front page needs to be updated as

there are gaps of lacking information, 16 of them thought that the notes that are

made to the evaluator need to be revised and probably made more concise, 41

were of the opinion that the section on the evaluation of readers, games and

charts should be maintained whereas 17 thought that the section needed to be

changed for better, seven think that the instrument is not long enough and needs

to be lengthened while 16 do not agree with that and think that the instrument is

too long and should be shortened. The remaining 52 respondents opted not to

express any opinion in this category of questions.

In conjunction with the earlier section on literature review, the aforementioned

reflect on the level of the instruments’ effectiveness, efficiency and impact. In

particular, it is to ascertain whether the instrument does enable the evaluators to

51

competently select only appropriate LTSM. That is whether it introduces

substantive changes in educators’ knowledge, skills and attitudes.

Another area of concern that was investigated by this study was whether the

instrument is used and if so, how often. Of the 157 respondents, 28 indicated

that they use the instrument once per year, six use it twice per year, 10 once per

quarter, 10 once per month, 15 more than once per month, 88 either do not use

the instrument at all or are not sure how often they use it.

4.2.4 Fourth theme: The Instrument encourages collaborative evaluation

One of the prominent ideas that were observed in this inquiry is that evaluators

of LTSM find it more sensible and more effective when to work collaboratively

with their colleagues. The model followed and adopted by the GDE is in line with

that situation at different levels where evaluation of LTSM takes place. The

setting for the evaluation process is not a natural one but is arranged and

organised by the evaluation coordinator whether at school, district or provincial

level. As stated in the earlier section on literature review, this kind of setup is in

compliance with what is asserted by Boulmetics and Dutwin (2000:2) that

evaluation happens mostly in controlled environments.

The findings reflect that 20% of evaluators who use the instrument rely on some

assistance from others. Although this does not form a majority of the

respondents, there is evidence that social interaction is the way in which

evaluators prefer to work during evaluation. This is also evident as, Victor one of

the respondents stated that:

When you are with your peers you are able to give yourselvesautonomy, modify the instrument and convert the quantities toqualities. In this situation you are able to discuss with your

52

colleagues some issues that would not have been catered for by theinstrument (see Addendum F)

This correlates with what Jeremy from one of the focus groups says that: I thinkwe do a wonderful job when we work in our teams at school and atprovincial Evaluation Centres (see Addendum F). These perceptions are in

line with the view of Stake (in Boulmetics & Dutwin) in his transaction model

where he states that activity between the evaluator, organizational staff and

fellow evaluators is of significance.

The conclusion made above could have some relationship with the following

responses that were collected after the users of the evaluation instrument were

being asked about the complexity and usability of the instrument. In their

responses, 53 respondents thought that the instrument was easy to follow, 36

felt that it was difficult to use, 14 indicated that they do follow the way it should

be used provided they are given assistance, 6 thought that the instrument was

not usable at all, 47 were not sure or do not use it at all, and only one respondent

did not respond.

As illustrated in Section One of this document, at the Evaluation Centre

evaluators are divided into teams of three. This implies that every title will be

evaluated by three people. Each team comes together and discusses all the

relevant issues and sets up standards that will be considered in this process.

Each team member is therefore given a copy of the title and is required to spend

quality time reading the LTSM and using the instrument to perform evaluation.

Later on, the team meets and consolidates reports into a single composite

report. This is also in line with the constructivist conviction that meaning-making

activities of an individual do not take place in isolation but in a context that is

shaped by societal influences (Gravett 2001:20). Similarly, Vygotsky, the father

of constructivism, concurs that mental function is initially external (intermental)

and later on internal (intramental) (Wersch & Toma in Gravett 2001:20).

53

Secondly, at school level activities happen at Macro, Meso and Micro levels. At

the macro level the school and its community are participant; at the meso level it

is heads of departments and other committee heads; and at the micro level the

Learning Area educator. These school activities include evaluation and selection

of LTSM. An educator who is an evaluator finds him/herself involved in a

collaborative relationship. Evident from the focus group interview, it is clear that

evaluators would find evaluation a difficult and tormenting task if there was no

team work and collaborative construction of knowledge. This kind of encounter

is a collaborative inquiry (Wiessner and Mezirow in Mezirow 2000: 331).

4.3 SUMMARY

In this section, I conducted an analysis of the research findings using a

technique known as constant comparative analysis. Each of the findings was

placed as a category of its own and interrogated against the knowledge found in

literature review. Most importantly I illustrated how the responses obtained in the

focus group interviews contribute to the themes which lead to the conclusions

and recommendations that will be discussed in Section Five below.

SECTION FIVE

CONCLUSION AND RECOMMENDATIONS

54

In this inquiry, the aim was to establish the level at which GDE educators as

evaluators engage with the instrument used to evaluate and select LTSM. The

roots of the practice of evaluation and selection are based on the international,

national and local activities around the evaluation and selection of LTSM. In the

earlier section of this inquiry, data collection methods like the survey

questionnaire and focus interviews were used to gather the evaluators’

responses regarding the evaluation and selection process. The findings as

discussed in Section Four above will influence the conclusions and

recommendations to be made in this section.

5.1 CONCLUSION

Based on the findings one should be confident to make the assertion that

through the GDE’s utilization of the Evaluation instrument, educators as

evaluators, at different levels, are able to engage with the instrument and select

appropriate materials for their classrooms to a certain level of success.

However, the evaluation process is more meaningful when the educators engage

in some sort of a team work or collaboration which will also prompt mutual

assistance. For that to happen, the instrument used for evaluation is the core

element or artefact. It is also striking to learn that most educators are willing to

perform the function of evaluating and selecting LTSM as part of their work.

During the focus interviews, it was clear that the educators as evaluators were

gratified for the fact that the whole process of evaluation and selection posed an

opportunity for hem to engage with their pre-existing knowledge and experience.

The pre-existing knowledge might have been gathered when they underwent

training as educators, during In-Service-Training, or at their work stations. In

other words, they were given an opportunity to construct new knowledge utilizing

their pre-existing knowledge and experience by recognizing and reconstructing

experiences encountered on the social plane (Driver Scott in Gravett 2001:20).

The pre-existing knowledge and experience are pivotal in the LTSM evaluation

process. Through this inquiry, I have established that the GDE has, to some

55

extent, created a social plane that comes in the form of a partnership between

the LTSM coordinators and themselves. It has also been established that some

educators welcome the support and assistance that is offered by the GDE.

The evaluation and selection process would be impossible to pursue without the

existence of a curriculum-compliant, user friendly instrument. Evaluators of

LTSM find it easy to work with an instrument that is well-organised into sections.

The evaluators who have used the GDE instrument stated that the GDE

evaluation instrument does comply with this aspect and include technical,

professional and educational qualities (including compliance with the curriculum,

relevance to the Learning Area, relevance to the assessment requirements and

the level of the learners’ development (GDE 2003:7-15).

Furthermore, I conclude that the GDE should encourage evaluators to work more

collaboratively with their colleagues when performing evaluation. In the focus

interviews some of them expressed satisfaction with the model that the GDE has

adopted in which they work in teams. This may have increased the educators’

willingness to perform the evaluation process more regularly than they do

presently, even if it forms part of their day to day teaching obligations.

5.2 RECOMMENDATION

As part of the conclusion, it is important to make some recommendations that

may directly and indirectly benefit all the participants in the process of evaluating

LTSM. This includes evaluators, learning area educators, parents, school

governing body representatives, school management teams, GDE district and

head office LTSM units including any other person who is involved in the

evaluation and selection of LTSM.

Considering the level of commitment by educators in this process, I would appeal

to the GDE to be more conscious of the significance of this process. I recognise

the existence of the SLTSMCs which serve as some form of the GDE’s

56

recognition of the function. The manner in which passion for the evaluation of

LTSM is expressed by some evaluators prompts me to recommend that the GDE

formalizes the existence of the LTSM department in the schools’ organizational

chart. This would relieve the LTSM Educator or head from other activities that

may derail him/her from the focus of ensuring that quality LTSM are selected for

the schools.

As stated above, educators have a wish to perform the function more regularly.

This interconnects with the above-stated recommendation of formalizing the

LTSM structure at schools. Such an ideal would only possible if the latter is also

achieved. Furthermore, the GDE has introduced Gauteng-On-Line through

which Information and Computer Technology Infrastructure is installed at schools

according to a specific plan. The GDE needs to equip itself by composing

criteria for evaluation and selection of software as it is expected of materials

developers to approach the GDE in their bid to have their electronic LTSM

utilized in the schools’ classrooms. This would also assist the GDE to cope with

the possible development from printed to electronic LTSM. This would also

imply utilization of technologies as suggested by the Eastern Cape Department

of Education (2003: 17).

Lastly, in support of collaboration and in order to be sensitive to the evaluators’

need to work in teams, I would also like to recommend that the GDE, at the

Evaluation Centre, maintains the model of working in teams. If the model is

appropriately utilised, the process of evaluation and selection of LTSM could be

more effectively performed. By and large, one may conclude that the findings of

this inquiry have revealed that evaluators do need to be involved in collaborative

construction of meaning, using the evaluation instrument (which they regard as

useful). If LTSM evaluation is worthwhile, they are prepared to undertake it as

part of their daily duties.

57

LIST OF REFERENCES

Boulmetics, J. & Dutwin, Phylis (2000). The ABCs of evaluation – timeless

techniques for progam and project managers . San Fransisco; Jossey-Bass Inc.

58

Booth,W., Colomb,G. & Williams, J. (1995). The craft of research. In E.

Henning & S. Gravett (Eds), Perspective on lifelong learning/ Research and

writing composition. Melville:RAU Centre for Distance Education.

Brown, B.L. (1998). Applying constructivism in vocational and career education.

Information series, n. 378. Columbo. Ohio:ERIC clearinghouse on adult, career

and vocational education.

Caffarella, P.S. (1998). Planning programmes for adults. An interactive

process. Adult learning, 10(2).

Cervero, R.M. and Wilson, A.L. (1998). Reflecting on what program planners

do. In Cookson, P.S. (Ed.), Program planning for the training and continuing

education of adults (135-163). Malabar:Krieger Publishing.

Charmaz,K. (2002). Qualitative Interviewing and Grounded Theory Analysis . In

J.F. Gubrium and J.A. Holstein (eds), Handbook of interview research. Context

and Method. (679- 694). London:Sage .

Cookson, P.S. (Ed.) (1998). Program planning for the training and continuing

education of adults. North American Perspectives. Malabar:Krierger Publishing .

Czerniewicz, L. (1993) Learning lessons from African publishing. In Kromberg,

Govender, Birrell and Sibanyoni (Eds.), Publishing for democratic education (67-

80). Cape Town:Sached Trust.

Daloz, L.A.P. (2000). Transformative learning for the common good. In

Mezirow, J. 7 & Associates (Eds.), Learning as transformation:Critical

perspectives on a theory in progress. San Francisco:Jossey-Bass.

Denning, C. (1992). South Africa- Textbook sector study. International Book

Development Ltd: 1992.

59

Eastern Cape Department of Education (2004). Review of LSM supply chain).

Bisho: ECDE

Evans, L. & Davies, K. (2000). No sissy boys here: a content analysis of the

representation of masculinity in elementary school reading textbooks. In Sex

roles: A journal of research v42 n3-4 p255-70.

Engerstrom, Y. (1991). Activity theory and individual and social transformation.

Activity Theory 7(8), 6-17.

Fortuin, B. (2002). The learning experiences of first time e-learners. Melville:

RAU Faculty of Education.

Gauteng Department of Education (2003). Foundation Phase Evaluation

Instrument. Johannesburg: Gauteng Department of Education Head Office

Gauteng Department of Education (2002). Learning and Teaching Support

Materials policy. Johannesburg: Gauteng Department of Education Head Office

Gauteng Department of Education (2003). Limited Open Equivalent to Approval

policy. Johannesburg: Gauteng Department of Education Head Office

Gauteng Institute for Education Development – Gauteng Department of

Education (

Gravett, S. (2001). Adult learning:Designing and implementing learning events-

A dialogic approach. Pretoria:Van Schalkwyk.

Gravett, S. (2001). The design and implementation of learning events for

adults- Study Guide. Mellville:RAU Centre for Distance Education.

60

Henning, E., Van Rensburg, W.A.J. & Smit, B. (2004). Finding your way in

qualitative Research. Pretoria:Van Schaick Publishers.

Heinich, R, Molenda, M, Russel, .D. & Smaldino, S.E. (1996). Instructional

media and technologies for learning. Ohio:Prentice-Hall.

House, E. & Howe, K.R. (2003). Deliberative Democratic Evaluation. In

Kellagan, Stufflebeam & Wingate (Eds.), International handbook of Educational

Evaluation (79-103). Dordrecht:Kluwer Academic Publishers.

Kvale, S. (1996). Interviews. An Introduction to qualitative research

interviewing . London:Sage.

Lincoln, Y. (2003). Constructivist Knowing, Participatory Ethics and Responsive

Evaluation:A model for the 21st Century. In Kellagan, Stufflebeam & Wingate

(Eds.), International handbook of Educational Evaluation (69-78).

Dordrecht:Kluwer Academic Publishers.

Masokoane, G.U. (1993). Tackling racism in textbooks In Kromberg, Govender,

Birrell and Sibanyoni (Eds.), Publishing for democratic education (61-66). Cape

Town:Sached Trust.

Maykut,P. and Morehouse, R. (1994). Beginning qualitative research: a

philosophical and practical guide. London: Falmer Press

McKinney, C. (2005). Textbooks for diverse learners- A critical analysis of

learning materials used in South African schools. Cape Town: HSRC press.

Meloy, J.M. (2002). Writing the qualitative dissertation – understanding by

doing. New Jersey: Lawrence Earlbaum Associates Inc.

61

Menton, M & Hiebert, R.A. (1999). Learning disabilities research and practice.

Utah: University of Utah

Mezirow, J. (2000). Learning to think like an adult: Core concepts of

Transformation theory. In Mezirow, J. & Associates (Eds.), Learning as

transformation:Critical perspectives on a theory in progress. San

Francisco:Jossey-Bass.

Monyokolo,M. (1993). Financing and provision of textbooks in South African

Black Schools. Johannesburg: University of Witwatersrand

Oliphant, A.W. (993). Arguments for a strong, indigenous publishing industry.

In Kromberg, Govender, Birrell & Sibanyoni (Eds.), Publishing for democratic

education (67-106). Cape Town:Sached Trust.

Omolewa, M. & Kellaghan, T. (2003). Educational Evaluation in Africa. In

Kellagan, Stufflebeam & Wingate (Eds.), International handbook of Educational

Evaluation (465-479). Dordrecht:Kluwer Academic Publishers.

Potenza, E. (1993). It doesn’t make sense: educational standards in textbooks.

In Kromberg, Govender, Birrell & Sibanyoni (Eds.), Publishing for democratic

education (47-60). Cape Town:Sached Trust.

Proctor, A. & Monteith, M (1993). Textbooks, corruption and curricula In

Kromberg, Govender, Birrell & Sibanyoni (Eds.), Publishing for democratic

education (31-46). Cape Town:Sached Trust .

Phillips, N. & Hardy, C. (2002). Discourse Analysis. Investigating process of

social construction. Qualitative research methods series 50. London:Sage.

62

Scriven, M. (2003). Evaluation theory and Metatheory. In Kellagan,

Stufflebeam & Wingate (Eds.), International handbook of Educational Evaluation

(15-30). Dordrecht:Kluwer Academic Publishers.

Sork, T.J. (2000). Planning educational programmes. In A.L. Wilson & E.

Hayes (Eds.),. Handbook of adult and continuing education. San

Francisco:Jossey-Bass.

Serguin, R. (1989). The elaboration of school textbooks – Methodological

guide. UNESCO.

Shailaja, M. & Hiebert, E. (1999). Literature anthologies: The task for first-

grade readers. Michigan:

Sork, T.J. (1998). Program priorities, purposes, and objectives. In Cookson,

P.S. (Ed.), Program planning for the training and continuing education of adults

(273-299). Malabar:Krieger Publishing.

Stake, R. (2003). Responsive Evaluation. In Kellagan, Stufflebeam & Wingate

(Eds.), International handbook of Educational Evaluation (63-68).

Dordrecht:Kluwer Academic Publishers.

Stufflebeam, D. (2003). The CIPP Model for Evaluation. In Kellagan,

Stufflebeam & Wingate (Eds.), International handbook of Educational Evaluation

(31-62). Dordrecht:Kluwer Academic Publishers.

United Nations Commission for Human Rights (1998). Universal Declaration of

Human Rights:general Assembly Resolution 217(III), Article 26.

University of Johannesburg (2005). Academic ethics committee document for

discussion. Mellville: UJ.

63

Wiesener, C.A. & Mezirow, J. (2000). Theory building and the search for

common ground. In Mezirow, J. & Associates (Eds), Learning as

transformation:Critical perspectives on a theory in progress. San

Francisco:Jossey-Bass.

Wright, C. (1993). Racism and sexism:legal considerations. In Kromberg,

Govender, Birrell & Sibanyoni (Eds.), Publishing for democratic education (67-

80). Cape Town:Sached Trust.

ADDENDA

64

ADDENDUM A: EVALUATION INSTRUMENT

Evaluation Instrument for Learning Support MaterialsFor Foundation Phase

(REVISED NATIONAL CURRICULUM STATEMENTS)2004Learning area: Grade:

Publisher:Language:

Author/s :

Title: Number of components:

Subtitles: ISBN: Price:

Evaluators:Name Signature

Educators’ Guide Learners’ Book Work Book Reader(s)

Audiovisuals:Charts Poster Models Toys Puzzles Games

65

Team Leader(s)Name Signature

EVALUATION TOTAL TICK CROSSGeneric Evaluation- Section ALearning Area Evaluation – Section B

FINAL DECISION: Approve two ticks Approved � Not approved �

Date: ____________________

66

SECTION A GENERIC EVALUATION

Rating scale:Very little = 1 Partially = 2 Mostly = 3 Very much so = 4Circle the score that best describes the material/s.

1.Compliance with the RNCS Weight= 10%

1.1 The Critical and Developmental Outcomes are covered in the LTSM.1.2 All the Learning Outcomes of the Learning Area are covered effectively and comprehensively in terms of content and activities.1.3 The materials are in line with the Constitution in terms of sensitivity to issues of poverty, inequality, race, gender, age, disability and HIV/AIDS.1.4 The materials cater for learners with varied abilities, as stipulated in the policy on inclusion / Education White Paper 6.

110

16Score Total ×

1 2 3 4 1 2 3 4

1 2 3 4

1 2 3 4

2.Assessment Weight= 10%

2.1 The assessment activities accommodate diverse contexts, e.g. diverse cultures, environments and religions.2.2 The assessment activities allow for learners to integrate and apply knowledge and skills. 2.3 Different assessment methods have been used, e.g. self, peer, group, educator, etc to cover the various aspects of the learner’s performance.2.4 The Assessment Activities assist the learners to set their own goals for progress. 2.5 The material assess thinking skills at different levels

110

20Score Total ×

1 2 3 4

1 2 3 4

1 2 3 4

1 2 3 41 2 3 4

3. Integration of Learning AreasWeight = 5%

3.1 The principles of the RNCS such as social justice, healthy environment, Human rights etc. are inherent in the LTSM and integrated across various Learning Areas.3.2 The Learning Outcomes and Assessment Standards of different Learning Areas are integrated in the different activities.

15

8Score Total ×

1 2 3 4

1 2 3 4

67

Rating scale:Very little = 1 Partially = 2 Mostly = 3 Very much so = 4

Circle the score that best describes the material/s.

5. Progression and Relationship Weight = 5%

5.1 There is progression within and between activities eg. the activity on identifying dangers at home should precede a discussion on how to maintain safety at home.5.2 There is a meaningful conceptual relationship between activities.

15

8Score Total ×

1 2 3 4

1 2 3 4

6. Context Weight = 10%

6.1 The learner’s prior knowledge in terms of experience and context is sufficiently acknowledged as a starting point.6.2 The content and activities provide opportunities for learners to discover a world beyond their own.6.3 There is a balance between information familiar to the learners and new knowledge both in illustrations and in text.6.4 Provision has been made for recognition of various social structures, e.g. if the material only makes reference to Christian beliefs, learners whose families practice other religions will feel excluded.

110

16Score Total ×

1 2 3 4

1 2 3 4

1 2 3 4

1 2 3 4

4. Approach Weight = 5%

4.1 The learning and teaching strategies used in the LTSM are appropriate for the content i.e. there is a proper balance between pair, group, class and educator – led activities.4.2 There are various types of activities, e.g. discussions, debates, projects, etc to accommodate learners from different backgrounds, at different levels of development and with differing language proficiencies. 4.3 The various activities enhance learning and encourage learners to be active and constructive participants.

15

12Score Total ×

1 2 3 4

1 2 3 4

1 2 3 4

68

Rating scale:Very little = 1 Partially = 2 Mostly = 3 Very much so = 4

Circle the score that best describes the material/s.

7. Content Weight = 10%

7.1 The content is unbiased and free of prejudice. It should avoid gender stereotyping, racial inferences, offensive textual and visual material.7.2 The activities geared towards the attainment of the Assessment Standards correlate with the age and level of the learner. (The activities should be neither too simple nor too complex.) 7.3 There is a fair representation of different cultural groups in illustrations, names of characters and text.7.4 The content is accurate, up to date, coherent and conceptually sound.7.5 The content is based in the South African context and also including updated information where relevant.

110

20Score Total ×

1 2 3 4

1 2 3 4

1 2 3 4

1 2 3 41 2 3 4

8. Relationship between content, illustrations and activities Weight = 5%

8.1 The activities are sufficiently related to the text and/or illustrations that coincide with them.8.2 Content, illustrations and related activities address the same outcome. However, it is possible that the activities incorporate other Learning Outcomes through the principle of integration.8.3 The activities prompt the learners to participate in different ways to avoid monotony, e.g. a reading text should not always be followed by questions to be answered orally.

15

12Score Total ×

1 2 3 4

1 2 3 4

1 2 3 4

9. Language and Style Weight = 10%

9.1 The vocabulary used in the materials is clear, accessible and appropriate for target group. 9.2 New concepts and information are introduced in a clear and logical manner. 9.3 The language used does not discriminate against certain races. 9.4 The language used is sensitive to issues of gender. 9.5 The language used does not denigrate certain religions and cultures. 9.6 The language used does not encourage stereotyping. 9.7 The language caters for learners with a variety of language backgrounds and abilities

110

28Score Total ×

1 2 3 4

1 2 3 41 2 3 41 2 3 41 2 3 41 2 3 41 2 3 4

Rating scale:Very little = 1 Partially = 2 Mostly = 3 Very much so = 4

69

Circle the score that best describes the material/s.10 Layout and Design

Weight = 10%

The page layout is suitable in terms of:

10.1 clear headings and sub-headings. 10.2 clearly marked activities. 10.3 suitable print size and type. 10.4 adequate white space and margins. The manner in which the design features are used:

10.5 is appealing and attractive to the eye of the learner. 10.6 draws attention and holds interest for learners to interact continuously with the material.

110

24Score Total ×

1 2 3 41 2 3 41 2 3 41 2 3 4

1 2 3 41 2 3 4

11.Pictures and Illustrations Weight = 5%

11.1 The pictures and illustrations are relevant to the activities being dealt with, i.e. that pictures and illustrations are not there as gap-fillers or merely for decorative purposes. 11.2 The pictures and illustrations contain all the details mentioned in the activities 11.3 The pictures and illustrations are representative of different races and genders as well as people with special needs, where necessary and appropriate.

15

12Score Total ×

1 2 3 4

1 2 3 4

1 2 3 4

12. Educator Support Weight = 15%

12.1 There is clear reference and cross-reference to all the components of the package.12.2 The educator is encouraged to adapt material to suit his/her individual situation andcontext

The educators’ guide gives:

12.2 Brief overview of the curriculum.12.3 Guidance to the educator on how to facilitate activities.12.4 Guidance to the educator on the application of different assessment methods.12.5 Solutions to problems.12.6 Guidance to educators to appropriate methods and learning strategies

115

28Score Total ×

1 2 3 4

1 2 3 4

1 2 3 41 2 3 41 2 3 41 2 3 41 2 3 4

TOTAL SECTION A SCORE:

1. Add the scores to get a total

70

2. Write this score for section A on page 1

ADDENDUM B: TABLES AND GRAPHS

71

TABLE 1 Categories as determined by the focus group interviews (modified and

adapted from Maykut and Morehouse 1994)

CODES POSSIBLE CATEGORIESEvaluationhappens

with someassistance(independe

ntly)

Instrument is

the core

element for

evaluation

process

Instrument

encourages

collaborative

evaluation

Willingness among

educators to perform

the function as part of

their work

Inadequate support

to institutions

X

Instrument used by

different sections in

the GDE

X X

Instrument well

marketed within the

GDE

X X

Effective training

sessions

X

Instrument regularly

used

X

User friendly

instrument and well

structured

X

Instrument

addresses the needs

(including curriculum)

X

Instrument enables

the educator to

decide on LTSM

X

Instrument should be

improved

X

Instrument used in

collaboration

X

72

Instrument boosts

confidence in

evaluation

X

Evaluation is

indispensable

X

Table 2. Suggestions made by evaluators on changes that the GDE should

consider to effect on the instrument used for evaluation

Category Tallies Frequency

73

Update information on

front page of instrument

IIII III

08

Revised section on Notes

to the evaluators

IIII IIII IIII I I6

Maintain section for

evaluating readers,

games and charts

IIII IIII IIII IIII IIII IIII IIII

IIII I

41

Change section for

evaluating readers,

games and charts

IIII IIII IIII II 17

Lengthen instrument IIII II 07

Shorten instrument IIII IIII IIII I 16

No response/ no

changes suggested

IIII IIII IIII IIII IIII IIII IIII

IIII IIII IIII II

52

TOTAL 157

74

0

10

20

30

40

50

60

NUMBER OF RESPONDENTS

UPDATEFRONT

REVISE NOTES MAINTAINSECTIONS

CHANGESECTIONS

LENGTHENINSTRUMENT

SHORTENINSTRUMENT

NO RESPONSE

S1

RESPONSES

Graph 1: IMPROVEMENTS AS SUGGESTED BY EDUCATORS ON THE GDE EVALUATION INSTRUMENT

75

Table 3: Evaluators’ indication of how frequent they use the evaluation

instrument

Category Tallies FrequencyInstrument used once per

year

IIII IIII IIII IIII IIII III

28

Instrument used twice

per year

IIII I 06

Instrument used once per

quarter in a year

IIII IIII 10

Instrument used once per

month

IIII IIII 10

Instrument used more

often than the above

IIII IIII IIII 15

Educators do not use the

instrument or are not

sure how often they use

it

IIII IIII IIII IIII IIII IIII IIII

IIII IIII IIII IIII IIII IIII IIII

IIII IIII IIII III

88

TOTAL 157

76

28

610 10

15

88

0

10

20

30

40

50

60

70

80

90

NUMBER OFEDUCATORS

ANNUALLY SEMESTERLY QUARTERLY MONTHLY MORE NONE

FREQUENCY

Graph 2: EDUCATORS' UTILIZATION OF THE GDE LTSM EVALUATION INSTRUMENT

77

Table 4: How educators view complexity of using the LTSM evaluationinstrument

Category Tallies Frequency

Instrument easy to follow IIII IIII IIII IIII IIII III IIII

IIII IIII IIII III

53

Instrument difficult to use IIII IIII IIII IIII IIII IIII IIII I 36

Follow with assistance IIII IIII IIII 14Instrument is not usable

at all

IIII I O6

Educators not sure or do

not use the instrument at

all

IIII IIII IIII IIII IIII IIII IIII

IIII IIII II

47

No response I 01

TOTAL 157

78

0

10

20

30

40

50

60

NUMBER OF RESPONDENTS

INSTRUMENT COMPLEXITY

Graph 3: EDUCATORS EXPOSITION OF INSTRUMENT COMPLEXITY

EASY DIFFICULT ASSISTANCE NOT USABLE NO RESPONSE

79

ADDENDUM C: TEACHERS AS EVALUATORS OF LEARNING ANDTEACHING SUPPORT MATERIALS

As stated in Section 1, this work is focused on the evaluation of Learning and

Teaching Support Materials (LTSM) as opposed to textbooks. The reason is

centreed on the fact that textbooks are not the only but one type of LTSM.

However, other types of LTSM, like educators’ guides, learners’ guides and

workbooks, may be regarded as forms of textbooks. As specified by Davies

(2000:126), textbooks, in whatever form, are aimed at fulfilling three main

functions, namely ‘interpersonal’ (relationship between writer and reader),

‘ideational’ (selection of information) and textual (constructing a coherent

message). Evaluators, by means of evaluation instruments endeavour to

establish whether these functions are fulfilled by LTSM.

As required by the National Department of Education through the Revised

National Curriculum Statements (RNCS), publishers develop their LTSM in the

form of a submission which commonly consists of a Teacher’s Guide, Learners’

Work/ Activity book, and Reader. As implied earlier on, LTSM also include

puzzles, charts, audiovisual materials, games, magazines, newspapers and

another wide variety of materials. The aim of this study is to assess the

usefulness of the evaluation instrument that is used for selection of LTSM. An

inquiry will be made on how educators engage with the instrument.

Rationale for Evaluation of Learning and Teaching Support Materials

As a starting point, one needs to consider the meaning of the word ‘evaluate’.

The concept is synonymous with words like ‘assess’, ‘appraise’, ‘weigh’,

‘consider’,’ review’ ‘gauge’, ‘calculate’ ‘moderate’, ‘adjudicate’ ‘referee’ ‘umpire’

and ‘estimate’. Interestingly, the common goal or intention implied in all the

aforementioned terms is to find the value, worth, or usefulness of the subject.

This is what evaluators of materials are conscious about when evaluating LTSM.

To be more precise, evaluation of LTSM entails objective and intensive reading,

80

bearing in mind criteria as stipulated in an evaluation instrument; writing down

analytic points (strong and weak to determine whether the LTSM are

educationally sound); scoring and writing qualitative reports/ reviews about every

component of a publishers’ LTSM.

Commonly, upon completion of the evaluation process, education departments

select LTSM that meet specified minimum requirements and place those in a list

of approved LTSM. The list is eventually converted into a catalogue. GDE

names that ‘List of Approved Materials’. It is important to state that evaluation

occurs at different levels:outside the school at the Evaluation Centre and at the

school by individuals and teacher teams (e.g. LTSM committees within the

school). This implies that schools have a role to play in the selection of LTSM

considering the needs of their teachers, learners and the surrounding school

communities.

This serves as a complement for what schools and school communities agree on

as determined by and as a result of planning that is done at Macro (including

school and its community members), Meso (at the level of Heads of Departments

and their departmental educators) and Micro (involving the grade and learning

Area educator) level.

The objective for LTSM Evaluation and Review is to make a positive impact on

the effectiveness and efficiency of its provisioning, defined as having the right

materials at the least cost in the right institutions at the right time (Eastern Cape

Department of Education, 2004). In order to perform the task of evaluation of

LTSM successfully, the evaluator needs to be given preset criteria in order to

ascertain the parameters within which to function. The criteria are set up by the

relevant education department, in this regard Gauteng Department of Education

(GDE) through its LTSM unit. The criteria come in the form of a grade or Phase

– specific evaluation instrument that is adapted to the needs of the curriculum,

i.e. the Revised National Statement.

81

However, it is important to note that evaluation of LTSM is not the only activity in

an education departments’ process map but forms an important and

indispensable area. This is evident in Figure 1 and 2 as specified by models

used by the Eastern Cape Department of Education and GDE respectively.

Figure 2.1. Generic Building Blocks of a Supply Chain (The Eastern Cape

Education Department Model) Key Stages Identification of

requirementSelection andprocurement

Delivery & payment

Typical Building Blocks Assessing Stock in hand Identification ofsuitable suppliers andmaterials

Production

Determining requirementquantity and quality

Determining price Delivery

Assessing required levelof supply

Selection of price Distribution

Assessing resourceavailability

Ordering Payment

Inter-dependent OperationalDimensions

Typical People IssuesWho? Are they capable and willing? Are their mandates clear? Typical Systems and Process IssuesAre processes sufficient, cost effective and correctly sequenced? Arethey integrated? Do system arrangements support effective and efficientsupply and prevent leakages Typical Technology issues

Are appropriate technologies in place to support processes? Are they

used? Can they be improved?

Curriculum changes have necessitated provincial education departments to

replace LTSM that schools have been using and that has also prompted LTSM

evaluation and review processes that are provincially controlled. Gauteng

Department of Education is no exception to the rule. In order to accomplish the

task, provincial education departments have to design a criteria-bound and

curriculum compliant evaluation instruments. In view of the narrow gap between

82

the grades, the GDE designed an inclusive Foundation Phase and Intermediate

Phase evaluation instruments. However, due to complexity in the subsequent

Phases and grades and also the implementation plan of the national Department

of Education, a grade specific instrument may have to be designed and

implemented per grade. This has been the case with instruments that were

designed for curricula prior to the RNCS.

83

SendPODs forGDED/OEdusolutions forpayment

LTSM – Process Map

PROCESSReviewDevelop

material

(Proposal)

Evaluation

Expo

Requisitioning-GDE601 forstationery-GDE600 fortextbooks

Orderingbysection21.1.cschools(from theapprovedlist)

CapturingOrderingfor Non-section21.1.c

Orderingbysuppliers

Requestforprintingof LTSM

Printing(textbooks)Manufacturing(stationery)

Deliverytoschools

Distribution ofLTSM

Retrievalprocess

Schoolspayinvoiceas perPOD

RESPONSIBLEPERSON

DistributorsServiceProviders(SMMEs)

SLTSMCSGBEducators

LTSMEducators/SLTSMC

Printers/manufactures

Publishers –stationerytenderers

TextbooksuppliersStationerytenderers

GDED/O orServiceproviders

Section21.1.cschools

Non-Section21.1.cschools

H/OD/O

GDE(H/O)

GDE(H/O) orServiceProvider

PublishersAuthorMaterialdevelo

LTSM(manuscript)

List ofapproved/ RejectedMaterials

Individual reviewguide

Requisition (toD/O orServiceproviders)

Orders(tosuppliers)

Orders/sundry (tosuppliers)

Orders(topublishers)

PrintedLTSMandstationery

POD toschools

DistributedLTSM

Retrievedmaterial

Deliveryschedule

Procurement, Monitoring, Control and Reporting Stages (by EduSolutions) PRE-Procurement Stage (Evaluation and Selection)

Figure 2.2: The GDE LTSM Process Map

84

PRODUCT/OUTPUT

85

EXPO-samples-price list-Catalogue

PrintedandmanufacturedLTSMforschools

What is expected of an educator as an evaluator of LTSM? (An ideal LTSMEvaluator)

Based on the above, one may argue that the task of evaluating LTSM is

extremely significant as it is a means to ensure that the design features of the

National Curriculum Statement ( viz:Critical Outcomes, Assessment Standards

Learning Outcomes) are achieved. This would assist in ensuring that South

Africa ends up with competitive, world-class citizens.

The above-mentioned implies that education departments are compelled to take

evaluation as an indispensable, vital function. Logically it implies that the GDE

has to ensure that anyone who is given the task possesses the appropriate skills,

experience and attitude. This means that the process of selecting evaluators

has to be accurately executed. The GDE uses largely educators as evaluators

of LTSM and criteria that are used as a measure include, among other things,

prior experience, Learning Area experience and experience of teaching in the

grade.

As stated above, prior evaluation experience is considered when selecting

evaluators of materials to select materials for the entire Gauteng province. This

is based on the fact that the GDE is faced with the task of training all the

selected evaluators and, it goes without training that it is much easier to train a

group in which the majority of the participants have been trained before. Some

of the experienced evaluators are given positions of more responsibility at the

Evaluation Centre e.g. they are selected as chief evaluators. Besides for the

aforementioned, the likelihood is that the experienced evaluators will perform the

function better than a novice evaluator.

Likewise, the teacher’s Learning area/ programme experience is also regarded

as a vital criterion in the selection of an evaluator. All educators are expected to

have some experience in teaching in general. This refers to the so called

traditional or NATED-550 curriculum. However, other teachers have not had

86

training on contemporary curriculum issues ranging from the time of Curriculum

2005 to the current era of Revised National Curriculum Statements. Teachers

who have been trained and have taught in the learning area/ programme are a

preference. However, that does not mean that the rest will be marginalized but

they are given some special kind of attention in order to make up for their

inexperience should they be utilized. The same considerations are given with

the experience in teaching the grade.

87

ADDENDUM D: APPLICATION TO BE LTSM EVALUATOR

GAUTENG DEPARTMENT OF EDUCATIONCHIEF DIRECTORATE:CURRICULUM AND PROFESSIONAL DEVELOPMENT AND SUPPORT

APPLICATION TO BE APPOINTED AS AN EVALUATOR OF LEARNING AND TEACHINGSUPPORT MATERIALS (LTSM) AND LIBRARY RESOURCE MATERIALS (LRM)

PLEASE NOTE

1. Only the following persons may apply• Educators presently teaching in the Foundation Phase• University lecturers with OBE / NCS knowledge• Consultants of NGOs currently involved in OBE activities• District and Head Office Learning Area / Programme facilitators• Educators with experience in evaluation of Library and classroom materials

2. Applications must be submitted to the relevant District Office:EducationResources LSM facilitator by 25 June 2003

3. Please attach a certified copy of:• Your qualifications (Including Matriculation Certificate and Statements

of Results)• Your most recent salary advice reflecting the date (if employed by the

GDE)• A Review Report of a book you have recently read and evaluated

4. The selected applicants must be available for all meetings (even onWeekends)

Please complete only one character / number per block, or mark the appropriate block with an “X”

PERSONAL DETAILS

Identity number

PERSAL NO

Title Initials

88

First Names

Home Language

Surname

Postal Address

Postal code

Telephone Number (work)Telephone Number (Home)Fax NumberCell Number

Preferred Language for CommunicationEnglish

Afrikaans

ACADEMIC DETAILS (Completed Degrees / Diplomas only)

Academic QualificationsDegree / Diploma Institutions Major subject (s) Year completed

Professional / Technical QualificationsDegree / Diploma Institution Major subject(s) Year completed

Institution where employed

89

Centre Number

Present post held (e.g. Educator. HOD. Principal)

TEACHING EXPERIENCE

NAME OF SCHOOL INSTITUTION

POSITION HELD

SUBJECTS /LEARNING PROGRAMMES(AREAS) TAUGHT

GRADESe.g.1,2,3

PERIOD

FROM TO

EXPERIENCE IN EVALUATIONLEARNING AREA/ PROGRAMME / SUBJECT IN WHICH YEARS?

LEARNING PROGRAMMES OF CHOICE FOR EVALUATION (State three in order preference)

TEACHINGEXPERIENCE IN YEARS

1ST

LanguageofLearningandTeaching.

2nd

LanguageofLearningandTeaching

1.2.3.4.5.

I CERTIFY THAT THE ABOVE INFORMATION IS CORRECT AND ACCURATE AND ACKNOWLEDGE THAT ANYERRORS OF FALSE DECLARATION IN THIS APPLICATION WILL LEAD TO ITS DISQUALIFICATION.

_____________________________ ________________________ ____________________SIGNATURE OF APPLICANT PRINT NAME DATE

C. DECLARATION

90

1. THE PRINCIPAL /UNIT HEAD OF ________________________ SCHOOL/ OFFICCEDECLARE THAT ALL THE INFORMATION IN THIS APPLICATION IS CORRECT TOTHE BEST OF MY KNOWLEDGE AND THAT THE APPLICANT IS QUALIFIED IN ALLRESPECTS TO EVALUATE MATERIALS FOR THE LEARNING PROGRAMMEAPPLIED FOR.

NAME OF PRINCIPAL/ UNIT HEAD__________________

SIGNATURE OF PRINCIPAL / UNIT HEAD_________________

_________________________

CONTACT TEL NO.

__________________________DATE

SCHOOL DATE STAMP

________________________

BANK DEPOSIT DETAILS OF APPLICANT

NAME OF APPLICANT:_______________PERSAL NO:____________________NAME OF ACCOUNT HOLDER:_________________BANK:_________________________BRANCH:________________________ BRANCH CODE:_______________ACCOUNT NO:______________________TYPE OF ACCOUNT:____________________________

(Current, Transmission, Cheque, Savings etc)

For District Use Only

I hereby verify that:

1. The applicant is currently teaching in the Phase2. I recommend / do not recommend the applicant to be selected as evaluator.

NAME OF FACILITATOR/ DISTRICT LSM CO-COORDINATOR:_________________________________________-

SIGNATURE:_____________________DATE____________________

For Head Office Only

91

Application Accepted / Not Accepted

NAME OF PANEL MEMBER

SIGNATURE:_________________________

DATE:_________________________

92

ADDENDUM E: ABSTRACT FROM GDE LOETA POLICY DOCUMENT

GAUTENG DEPARTMENT OF EDUCATION

UMnyango WezeMfundo Lefapha la Thuto

Department of Education Departement van Onderwys

Draft LOETA PROSUREMENT SYSTEM POLICY

A. BACKGROUND

Prior to 1997, the GDE used a system to select Learning and Teaching Support

Materials (LTSM) on behalf of the educators in which educators were not involved. The

GDE realised that the system had some shortcomings and in order to rectify that

developed a system referred to as the ‘Open system’ in which educators had a role to

play in selecting LTSM. The ‘Open system’ implied that review, selection and evaluation

of LTSM took place at various levels (including school level). In addition, and also in line

with the principle of Self-Managing Institutions as rooted in the South African Schools’

Act, the ‘Open system’ allowed schools to commence to manage their own budgets,

make own choices on stationery (as opposed to pre-packs) and select textbooks that

are presented during the publishers’ marketing drive process in the form of exhibitions,

Schools-to-School visits and other means (i.e. from the open market).

Furthermore, the GDE introduced a Review Guide that is in the form of a collection of

review reports and which serves as an optional aid to the educator. The GDE therefore

stopped issuing a catalogue or any type of LTSM list for schools. It was hoped that

every educator would be trained on how to evaluate and select LTSM independently.

Such training would be backed by exhibitions and displays of publishers’ LTSM. The

main tool to assist the educator in evaluating the LTSM would be the Evaluation

Instrument.

93

In view of the above-stated shortcomings, the ‘Open system’ was put under debate to

subject it to observation, evaluation and review to test for its suitability, compatibility,

relevance and usefulness. Consultation was done with various Head Office and District

units (in particular, LTSM and Curriculum units). These debates were also backed by

comparative work that was done with some provinces. The outcome was the

replacement the system with a ‘Limited Open (equivalent to approval) System’ (LOETA

system). In the system, the educators still have an active role to play but have to select

their classroom materials from a list of approved LTSM. This also implies that the GDE

will now take a decision on the criteria for selection based on the curriculum

requirements, decide on a number of submissions per Learning Area/ Programme,

produce a list of approved LTSM and advise the educators accordingly. Only materials

that were approved will be exhibited, procured and distributed to the schools.

94

B. LTSM PROCESSES AND CONCEPTS WITHIN THE LOETA SYSTEM

1. DEVELOPMENT OF LTSM

This is the initial step in the LTSM process map. Different persons including the

Learning Area educator, District and Head Office officials and the publishers’ authors

do LTSM development. The GDE also has an important role to play in the

development of educators on how to develop materials. This assists in preparing

the educators for the evaluation, selection and review and utilization of LTSM

processes that will be explained below. This step is characterised by, among other

things, the actual writing of printed materials by authors, development of educational

equipment (e.g. puppets, toys, puzzles etc). However, it is important to note that

the emphasis of the procurement process is to be discussed hereunder.

2. SUBMISSION OF LTSM BY PUBLISHERS

Once the development of LTSM discussed above concludes, the publishers desire to

supply schools with their materials (i.e. they require the GDE and its schools to

procure the LTSM). As a pre-requisite the GDE needs to evaluate and possibly

review the LTSM prior to the procurement process. Every year, the GDE sets a date

on which all the publishers are required to submit LTSM that they desire to supply to

the schools.

Coupled with the submission date is the Submission Specification memorandum.

The Submission statistics include Typing specifications (font size, font style, program

(e.g. MS Excel) and specification on software to be used), Packaging specifications

(sorting by language and learning area, the contents and number of sets per

submission- normally three and cover details to be stated on the cover page per set.

Furthermore, the specifications memorandum specifies the submission fees, which

are normally charged per submission – The GDE uses its discretion whether to

charge per submission or per Learning Area/ Programme depending on a variety of

variables including the total number of submissions, number of implementing

grades, etc. The submission amount will always be reviewed and decided by the

GDE on an annual basis. The funds are credited to the Gauteng Education

Development Trust account. These funds are used to cater for the professional and

administrative costs of the evaluation process.

3. EVALUATION, REVIEW AND SELECTION OF LTSM

95

The three processes of Evaluation, Review and selection of LTSM as defined by the

GDE are interdependent processes and serve as a pre-requisite for LTSM

procurement. Depending on a number of possible causative factors, the GDE

possesses the right to use its prerogative in deciding whether or not to undertake

any one of the three steps. However, the definition below is meant for the purpose

of indicating the common understanding of the concepts among all users and

participants within the GDE

EVALUATION OF LTSM

During the Evaluation stage, LTSM that have been received from the publishers are

evaluated for appropriateness for the grade or and for usability in the classroom.

Panels of evaluators who have deep knowledge of the curriculum, who have/ are

ready to implement it and are competent in using evaluation instruments and

statistically designed score sheets are carefully selected and appointed on a

temporary basis. These evaluators are selected from the larger South African

community of professionals and may include school educators, education

practitioners, higher education educators/ lecturers and others.

The publishers’ materials are scored using a quantitative approach by means of a

score sheet. Materials that achieve more than the minimum score are put on an

‘Approved Materials List’. In the ‘Approved Materials List’, the choice will still be

wide, as a number of titles per Learning Area will be listed. Materials with minimal

errors are put on a ‘Conditionally Approved Materials List’ and the publishers are

given comments on the materials’ weaknesses and a re-submission date. Materials

that are far less than the minimum are put on a ‘Rejected List’ and are totally not

considered. However, the GDE may announce an appeal date during which

publishers whose materials have been rejected may appeal. The date should be

seven days after the publishers have been advised of the rejection of their materials.

A reasonable amount of time will be given for the publishers to rectify/ rewrite their

materials and resubmit. Announcement will then be made to all the publishers and

the final ‘Approved Materials List’ is produced and locked. The locking of the system

implies that no further entries will be added or removed from the list. Hard copies

and write-protected CDs will be sent to all the GDE institutions and offices prior to

the Exhibition period.

96

3.2 REVIEW OF LTSM

The Review process is characterised by the use of mostly qualitative Evaluation

instruments to write up reports on reviews done on materials. Only materials that were

evaluated in the last stage and put on the ‘Approved Materials List’ will be reviewed.The review reports are done at different levels:first by individual reviewers, secondly the

review reports are put together into a Composite report. The composite report is

checked and verified by the team leader, coordinator and submitted to the GDE Quality

assurance team. Once found to be of a reasonable quality the reports are listed and put

together into a compilation known as the Review Guide. The Review Guide is released

in both the form of a hard copy and CD.

The Review of LTSM is not only centrally done but is also done at schools level for

various reasons at different intervals. The schools educators and School Learning and

Teaching Support Materials Committees (SLTSMCs) are invited to exhibitions/ EXPO

that are organized by Head Office and District Offices. During the exhibitions the

educators are given the opportunity to view publishers’ materials; use their Evaluation

Instrument and Review Guide to read more on the materials that have been reviewed;

and select materials that they desire to procure for their schools. However, that is not

the final step, as the SLTSMCs will also nominate a selection committee that will use the

Evaluation Instrument to select from the ‘Approved Materials List’. Furthermore, the

school may suspect some materials to be outdated or obsolete. Before deciding on

transfer of the materials they need to use the Evaluation instrument and compile a

report stating why the materials are declared obsolete.

3.3 SELECTION OF MATERIALS

All the participants do the selection of LTSM at the schools level. These participants

include grade educators, Learning area/ programme educators, Heads of Departments,

the School Management teams, the SLTSMCs and the School Governing Bodies (SGB).

Once a decision has been finalized, the SLSMC puts together requisitions and orders

that will be explained in the next sections of this document.

97

4. EXHIBITIONS

In order to give the publishers’ materials maximum exposure and to ensure that all the

educators are aware of and explore the LTSM that are available in the market, the GDE

organizes exhibitions or LTSM EXPO. The EXPO is held at a central venue for a

blocked period of about seven days. The advantage of a central EXPO is that

publishers do not have to move on a daily basis from district to district. That saves on

valuable time and minimizes expenses. Furthermore it will enable publishers to bring a

wider variety of available materials across the grades. Monitoring the EXPO becomes

more efficient and sufficient.

During the EXPO, over and above the display of materials, seminars are held. Topics of

educational value are to be discussed. Presentations are to be done by various

stakeholders including publishers, suppliers, distributors, service providers, higher

education institution academics, independent intellectuals, publishers’ and bookseller’s

associations, education officials and officials from other state departments. Stalls will be

hired out to the exhibitors at a fee negotiated with all the stakeholders and an events

management service provider will be appointed to market, organize and finalize the

event.

98

5. REQUISITIONING AND CAPTURING The requisitioning process takes place both in Section 21.1.c and Non- Section

21.1.c schools. However, the process happens at two different levels at both the

school types. In Section 21.1.c schools it is an internal schools process in which

departments or Learning Areas requisition what they require to the SLTSMC who in

turn forward the requisitions to the School Management Team (SMT) and the SGB

for final approval.

In Non- Section 21.1.c schools the process referred to above initiates inside the

school but is concluded at the District Office. Once Requisitions are recommended

and signed, Non-Section 21.1.c schools forward the relevant forms to District office

or service provider (if appointed as this is regarded as a Non-Core function and the

GDE may decide to outsource the function). The District office/ Service provider

therefore scrutinizes, checks, verifies the forms for correctness and to ascertain

whether the schools have not exceeded the budget. After that the District office/

Service provider captures the requisitions on a specific/ compatible accounting

system. Once capturing is done, the next step, ordering takes place.

6. ORDERING

As mentioned in 4 above, Section21.1.c schools prepare and place their orders

directly with textbook distributors/ suppliers and stationery suppliers. The

Section21.1.c schools together with the distributors and suppliers need to sign terms

of agreement to ensure that deliveries and payments are done as per the

expectations of both the parties. Suppliers therefore place their orders with

publishers

As stated above, district offices/ service providers place orders on behalf of Non-

section21.1.c schools with textbook suppliers and stationery tenderers. In the

absence of the latter orders are placed with the service providers who in turn

place the orders with publishers. Upon receiving the orders publishers up

together composite orders to request the printers to perform the printing works.

The average duration the print run is eight weeks after which the printers deliver

to the publishers. In the case of stationery orders are placed with stationery

tenderers who in turn place the orders with stationery manufacturers. The

99

stationery manufactures are therefore expected to deliver directly to the

tenderers.

7. PRINTING

The printing of media/ printed LTSM (books, textbooks) is a lengthy period and

the GDE considers that in determining delivery dates. Publishers place an order

with printers. Averagely an allowance of eight weeks is given for the process.

On conclusion of printing, the printers and manufacturers deliver LTSM.

8. DELIVERIES AND PAYMENTS

The deliveries process presents itself as a reversal of the ordering process.

Printers/ stationery manufacturers deliver LTSM to the publishers who in turn

deliver to the suppliers/ distributors/ service providers who also deliver to

schools. When deliveries are done to schools, a delivery advice/ Proof of

Delivery (POD) is signed and stamped by the principal of the school. A copy is

left for the school’s records.

Schools are always urged to check the contents of the deliveries and reconcile

those with their original requisition forms. Non- Section 21.1.c schools are

required to present copies of their PODs to the District office or appointed

service provider for payment purposes. Section 21.1. c schools, upon checking

and finding the LTSM satisfactory, process and make the payment to the

suppliers/ distributors. The service provider/ district office will also pay the

supplier/ distributors per POD.

9. STAKEHOLDERS’ PARTICIPITATION

100

The GDE has established a forum in which all the stakeholders are invited to

make educationally sound contributions to the GDE education system. The

stakeholders include, among others, suppliers of stationery, distributors of

textbooks, publishers’ associations, booksellers associations, independent

publishers, academics, educators, etc. A stakeholders’ code of conduct has

been developed in consultation with stakeholders and is subject to public

scrutiny.

The purpose of the code of conduct includes managing and controlling activities

of independent suppliers and distributors some of whom access the GDE

Section 21.1.c schools directly. All the stakeholders are required to sign and

acknowledge the existence of the code of conduct. Stakeholders’ Forum

meetings are arranged for and coordinated by the GDE in consultation with

stakeholders.

101

ADDENDUM F: SURVEY QUESTIONNAIRE

SURVEY QUESTIONAIRE

NB:When completing this questionnaire, you are requested to:

• Please answer the questions with honesty• Answer all the applicable questions• Write legibly• Add whatever information that you deem important but was not asked for on

the margins• Avoid using your name as your identity is not required

Date:____________________

SECTION A:PERSONAL INFORMATION

Mark the appropriate block with an X or fill in the required information

Age (in years):

Gender: M F

Race: BLACK WHITE COLOURED INDIAN OTHER (specify)

Teaching experience (in years)

Location of school: Township Sub-urban Inner city Farm

SECTION B:EXPERIENCE IN EVALUATION

102

1. I have...Never seen the instrument beforeSeen but not used the instrumentOnly seen my colleagues use the instrument before Seen and used the instrument

2. I learnt about the instrument in the following way …

Through a workshop by the education departmentBy seeing my colleagues using it At the provincial Evaluation CentreFrom a close friend of mine who is a teacher

Note:Only respond to question 3 below if you have received training/ workshopby Gauteng Department of Education.

3. After receiving training on the utilization of the Evaluation Instrument I felt asfollows:

knowledgeable and confident to independentlyevaluate materialsCould try to evaluate but with a few doubts I have gained some knowledge but need more trainingAttending the training was not worthwhile

4. I have used the instrument intended for evaluating materials in the ….…..Phase

Foundation PhaseIntermediate PhaseSenior PhaseFurther Education and Training

5. I use the instrument …

Once per year Once in six monthsOnce per quarter in a yearOnce per month

103

More often than above

SECTION C:ACTUAL ENGAGEMENT WITH THE INSTRUMENT

1. The information that an evaluator is required to complete in the instrumentis….

SufficientAlmost sufficientLacks important detailsTotally insufficient

2. The way the procedure for evaluating materials is explained …

I can easily followI follow with some struggle I cannot follow without assistance from othersI cannot follow altogether

3. The instrument addresses the assessment qualities of the book…

SufficientlyVery wellModeratelyPoorlyInsufficiently

4. The way in which the instrument addresses the Teaching Approach used

by authors in their books (e.g. group, pair, class, and project)…

Focuses mostly on oneFocuses on twoFocuses on threeFocuses on all

5. The naming of the different sub-sections…

Does not capture all the relevant conceptsCaptures the relevant concepts to some extentAlmost captures all the relevant conceptsSufficiently captures all the relevant concepts

104

6. When using the instrument I…

Always find a repetition of some statementsOften find a repetition of some statementsSometimes find a repetition of some statementsNever find a repetition of any statements

7. The way the instrument is structured, I …

Am able to identify and eliminate inappropriate materialsI to a limited extent can differentiate between appropriateand inappropriate materialsI find it difficult to distinguish between appropriate andinappropriate materials I am unable to distinguish between appropriate andinappropriate materials

8. The instrument makes provision for

For some types of materialsFor all types of materialsOnly for prominent types of materials typesOnly for materials not intended for the said grade

9. If I were to improve on the instrument, I would do the following:

Update information on the front pageRevise the section called ‘notes to the Evaluator’ Effect changes but maintain sections for evaluatingreaders, games and charts Completely change the sections for evaluating readers,games and charts Lengthen the entire instrumentShorten the entire instrument

10.Overall, I think the instrument:

Is completely objective and fair to the publishersIs almost objective

105

Is almost subjectiveIs completely subjective and unfair to the publishers

ADDENDUM G: Interview transcripts

FOCUS GROUP INTERVIEWS

PLACE: ALBERTON TEACHERS CENTRE

DATE: 15 JULY 2004

CLUSTER NUMBER 1

RESEACHER: Colleagues, I would like us to take a discussion on the

process of evaluation and selection of LTSM. Precisely, the reason for me

to decide to select this particular group was based on your prior knowledge,

and experience.

JEREMY: Tell us exactly what areas you would like us to discuss.

RESEACHER: Oh well! This tells me that you have a lo to share with us.

What are the experiences that you have had as a participant in the process of

LTSM evaluation and selection?

RACHAEL: I know he has a lot to tell us because I was introduced by him

to the process. In fact, he increased my interest in the process of LTSM

evaluation.

106

JEREMY: It is actually a very complicated process when you are not given

support.

MAXWELL: Especially the so called District officials. At times I think

they are abdicating their responsibilities because they leave us without

support at all.

RESEACHER: What kind of support maybe?

SIZWE: For instance training us, giving us instruments and also feedback.

RESEACHER: Okay, I hear you... At what levels do the District officials

involve you as educators in the process?

HLOMU: They advise us to do that even at school level. In this case you

need to consult with other teachers via the LTSM committee to decide on

the kind of material to be selected for you classroom.

RACHAEL: Don’t forget the governing bodies. They will kill you!

JOAN: It is true they like to be involved but they need more support than us.

RESEACHER: Besides support from officials who else can give you the

necessary support.

107

JEREMY: I think we do a wonderful job when we work in our teams at

school and at provincial Evaluation Centres. Your colleagues give you

support, assurance and confidence.

RESEACHER: You have spoken about all the stake holders but have not

mentioned school management team members.

ELIZABETH: Forget about those. What they know is to hide away

circulars and memoranda that are sent by the department.

RESEACHER: Jeremy spoke about an Evaluation Centre. Do you think

that schools can do without having to go to the Evaluation Centre?

HLENGI: Yes we can because we do most of the work ourselves.

HLOMU: No! Those people give us a lot of guide lines. Without their

guidance our processes could not take off the ground.

SIZWE: In particular they supply us with the evaluation instrument.

Besides, teacher can be rowdy in the absence of authorities.

RESEACHER: How often do you think the evaluation process should take

place?

HLENGI: To be honest only when RNCS is introduced in a phase.

108

ELIZABETH: I definitely do not agree. I have been a team leader at the

centre and when we were trained it became evident that this process is

indispensable and needs to take place once per year.

RESEACHER: Do you get paid and how do you feel about that?

JEREMY: (pointing at Elizabeth) this one is supposed to support the

process because she scoops a lot of money during evaluations.

ELIZABETH: (In response) not necessarily for that purpose! Even if I am

not paid I would still do it. I actually think that it should be compulsory to

all the teachers.

HLOMU: You are right! Many of us think that the process would be more

effective if it takes place for several occasions in a year.

RACHAEL: Maybe they should establish a post for LTSM head of

department.

RESEACHER: Do you all think that the process should form part of your

work?

HLENGI: Over my dead body!

JEREMY: I think Rachael was right. This is a significant process for all the

stake holders in education.

109

RESEACHER: Thank you very much for availing yourselves to this

discussion. May I request you to avail yourselves to anyone who might

need an opportunity to utilize your knowledge and experience?

ALL: Thank you very much!

FOCUS GROUP INTERVIEWS

PLACE: ALBERTON TEACHERS CENTRE

DATE: 16 JULY 2004

CLUSTER NUMBER 4

RESEACHER: Colleagues, I would like us to take a discussion on the

process of evaluation and selection of LTSM. Precisely, the reason for me

to decide to select this particular group was based on your prior knowledge,

and experience. Do you think there is an improvement on the quality of

materials that are used in our classrooms and what is the cause?

BAFANA: Based on my experience as an educator, member of the materials

committee and evaluator of materials, I would think that the GDE is moving

towards the right direction. In the past we were excluded from the process

of taking decisions as to what kind of materials we wished to utilize to

develop our children. Materials that we didn’t desire were ordered on our

behalf. We were not given the opportunity to accept or reject. That was the

110

shame of Bantu education. We were forced to sign for and receive

undesired materials.

MARY: Although there are still some loopholes in the new system of

evaluating materials, we now feel as professionals who are respected as

opposed to being puppets.

RESEACHER: What kind of loopholes?

MARY: I am just hoping that there will be a time when the instrument that

we use for evaluation will assist the educator to select the materials that

don’t have any defects or weaknesses that could hamper or negatively affect

the teaching and learning processes in our schools. For now I think the

materials that we chose for ourselves still have some limitations.

ZANELE: I don’t squarely put the blame on the system that we use to

evaluate materials. The problem could be the non – availability of

appropriate materials from the publishing houses. Let us not blame the

department for poor materials in our systems but writers, developers and

publishers of materials.

ZANDILE: My worry is that the type pf materials that are available are

mainly text books. What about puzzles, charts, CDs, atlases, Dictionaries

and audiovisuals, etc.

RESEACHER: Do you think there is a special type of instrument to measure

the usefulness of the variety of materials?

111

BIGBOY: Presently there isn’t.

SANDILE: No Bigboy I think you are making a mistake when I was at the

Evaluation Centre, some publishers had submitted a wide variety of

materials including software. The challenge was to evaluate.

SIMON: What did you do?

ZANELE: I still recall our team was mainly given such materials and we

were directed to use a section of the instrument that was designed

particularly to evaluate such materials.

RESEACHER: Besides the actual evaluation, what did you enjoy or find to

be boring about being part of the evaluation group and to be at the

Evaluation Centre?

ZANDILE: The spirit of togetherness! To be taken care of by other

educators, listen to their advises , be criticesed, make a contribution,

develop ideas, write reports and await payment (laughing )

RESEACHER: Now that you have spoken about the payment of evaluators,

would you be willing to, at some point, participate in textbook evaluation

without remuneration?

SIMON: No ways, remember that as educators we receive meager funds in

the form of payments. I am therefore unwilling to work for no pay.

112

SANDILE: Remember that it is your calling. Besides, it is within the

principles of democracy that we should have a contribution to make in our

profession. We need to be proud by knowing that we were part of the

selection process. However, we will feel more secured and protected if we

have the backing and support of our line functionaries – in particular if our

principals could refrain from blockading the insemination of information .

RESEACHER: How could this is achieved?

SIMON: The best way is for the department of education to strengthen its

structures and also by means of consultative forums.

RESEACHER: I would like to thank you all for availing yourselves to this

short discussion.

ALL: You are welcome.

113

FOCUS GROUP INTERVIEWS

PLACE: ALBERTON TEACHERS CENTRE

DATE: 16 JULY 2004

CLUSTER NUMBER 3

RESEACHER: Colleagues, I would like us to take a discussion on the

process of evaluation and selection of LTSM. Precisely, the reason for me

to decide to select this particular group was based on your prior knowledge,

and experience. Who would like to start and share with us the

understanding of that process?

JOHNSON: That is a process that takes place when we are asked to go to

the exhibitions to represent our schools. We often go there on an afternoon

and are given samples of material to view and to go and share with our

colleagues at school. This is followed by the requisitioning of materials.

JOY: I don’t think the question relates to that.

114

RESEACHER: In fact the processes explained by Johnson do form part of

the evaluation and selection of LTSM. But the process starts at an earlier

stage.

PATRICK: It is when the department receives loads and loads of materials

and educators are requested to partake in the process of evaluating the

LTSM.

VICTOR: I have been to the Evaluation Centre and I do not wish to go back

there.

RESEACHER: Why?

VICTOR: Evaluators of materials work under severe pressure. There is no

time breathe. They divide you into teams and give you an unreasonable

amount of work.

WINNIE: Each team is given a minimum of ten submissions and is expected

to sit down and evaluate their ration without a break. Mind you this has to

happen over weekends.

MANDLA: Only three to four weekends.

RESEACHER: Do they assist you to do that.

JOY: Only by means of an evaluation instrument.

115

RESEACHER: Tell me about the instrument.

VICTOR: The instrument is grade – specific. It consists of questions that

are grouped according to themes and sub-themes. The questions are scored

according to either a three or five point scale. These questions assist the

evaluator on deciding whether or not a book is subjective, racist, gender-

insensitive, qualifies, and complies with the curriculum and so on.

SALLY: In addition the instrument has questions that deal with the physical

aspect. It is to test whether the book will last for a long period without

getting damaged. Also to determine whether the book is worth the price

that the publishers are asking for.

RESEACHER: Are you able to use that instrument without assistance?

VICTOR: If you have had some prior training you only need little

orientation to be able to cope with the task of evaluating materials for the

particular grade in the particular year.

KHAYA: Some of us have had an opportunity of being trained on several

occasions for different phases and grades. However, I have learned that

each year the instrument is up graded and is nearing perfection. That makes

it even simpler to use at the Evaluation Centre and at our schools.

SALLY: In fact I feel much freer to use the tool in the absence of

authorities. Under supervision, I feel threatened, tense and even lose

composure.

116

VICTOR: I also think so because when you are alone or at least with your

peers you are able to give yourselves autonomy, modify the instrument and

convert the quantities to qualities. In this situation you are able to discuss

with your colleagues some issues that would not have catered for by the

instrument.

RESEACHER: Do you think the department enables you as teachers to own

the process of evaluation and selection

JOHNSON: In the past the teacher was given very little opportunity to

partake in the process. We felt left out however, now we are given a certain

amount of control. When I was at the Evaluation Centre I wished that could

every month or at least once per quarter.

RESEACHER: I think we have covered a remarkable amount of the

intended outcomes of this participative discussion. May I thank you for

your time and active involvement in the process

117

FOCUS GROUP INTERVIEWS

PLACE: ALBERTON TEACHERS CENTRE

DATE: 16 JULY 2004

CLUSTER NUMBER 4

RESEACHER: Colleagues, I would like us to take a discussion on the

process of evaluation and selection of LTSM. Precisely, the reason for me

to decide to select this particular group was based on your prior knowledge,

and experience. Do you think there is an improvement on the quality of

materials that are used in our classrooms and what is the cause?

BAFANA: Based on my experience as an educator, member of the materials

committee and evaluator of materials, I would think that the GDE is moving

towards the right direction. In the past we were excluded from the process

of taking decisions as to what kind of materials we wished to utilize to

develop our children. Materials that we didn’t desire were ordered on our

behalf. We were not given the opportunity to accept or reject. That was the

118

shame of Bantu education. We were forced to sign for and receive

undesired materials.

MARY: Although there are still some loopholes in the new system of

evaluating materials, we now feel as professionals who are respected as

opposed to being puppets.

RESEACHER: What kind of loopholes?

MARY: I am just hoping that there will be a time when the instrument that

we use for evaluation will assist the educator to select the materials that

don’t have any defects or weaknesses that could hamper or negatively affect

the teaching and learning processes in our schools. For now I think the

materials that we chose for ourselves still have some limitations.

ZANELE: I don’t squarely put the blame on the system that we use to

evaluate materials. The problem could be the non – availability of

appropriate materials from the publishing houses. Let us not blame the

department for poor materials in our systems but writers, developers and

publishers of materials.

ZANDILE: My worry is that the type pf materials that are available are

mainly text books. What about puzzles, charts, CDs, atlases, Dictionaries

and audiovisuals, etc.

RESEACHER: Do you think there is a special type of instrument to measure

the usefulness of the variety of materials?

119

BIGBOY: Presently there isn’t.

SANDILE: No Bigboy I think you are making a mistake when I was at the

Evaluation Centre, some publishers had submitted a wide variety of

materials including software. The challenge was to evaluate.

SIMON: What did you do?

ZANELE: I still recall our team was mainly given such materials and we

were directed to use a section of the instrument that was designed

particularly to evaluate such materials.

RESEACHER: Besides the actual evaluation, what did you enjoy or find to

be boring about being part of the evaluation group and to be at the

Evaluation Centre?

ZANDILE: The spirit of togetherness! To be taken care of by other

educators, listen to their advises , be criticesed, make a contribution,

develop ideas, write reports and await payment (laughing )

RESEACHER: Now that you have spoken about the payment of evaluators,

would you be willing to, at some point, participate in textbook evaluation

without remuneration?

SIMON: No ways, remember that as educators we receive meager funds in

the form of payments. I am therefore unwilling to work for no pay.

120

SANDILE: Remember that it is your calling. Besides, it is within the

principles of democracy that we should have a contribution to make in our

profession. We need to be proud by knowing that we were part of the

selection process. However, we will feel more secured and protected if we

have the backing and support of our line functionaries – in particular if our

principals could refrain from blockading the insemination of information .

RESEACHER: How could this is achieved?

SIMON: The best way is for the department of education to strengthen its

structures and also by means of consultative forums.

RESEACHER: I would like to thank you all for availing yourselves to this

short discussion.

ALL: You are welcome.

121