Adversary-Oriented Evaluation (2)

SGDC 5023EVALUATION OF TEACHING

GROUP 5ADVERSARY-ORIENTED EVALUATION

APPROACHES

1

ARWINA SYAZWANI BINTI GHAZALI820403-11-5560

WAN ANISAH BINTI ABDUL KADIR810522-11-5572

NOOR FAZLINA YATI BT MAT RAZALI820208-11-5338

NORIHAN BT. ENDUT791223-11-5408

Prepared By:

2

ADVERSARY-ORIENTED EVALUATION APPROACHES

3

ADVERSARY-ORIENTED EVALUATION

Adversary-oriented evaluation is based on the judicial metaphor. It is assumed here that the potential for

evaluation bias by a single evaluator cannot be ruled out, and, therefore, each “side” should have a separate

evaluator to make their case.

4


Purposes of Evaluation:

Providing a balanced examination of all sides of controversial issues or highlighting both strengths and

weaknesses of a program.

6


Distinguishing Characteristics:

Use of public hearing, use of opposing points of view, decision based on arguments heard during

proceedings.

7

DEVELOPERS OF ADVERSARY-ORIENTED EVALUATION

APPROACHES AND THEIR CONTRIBUTIONS

8

GUBA [1965]

OWENS [1971]

WOLF [1973]

HISCOX & OWENS [1975]

WOLF [1975]

STENZEL [1976]

LEVINE [1976]

NORTHWEST REGIONAL

EDUCATIONAL LABORATORY [1977]9

DEVELOPERS OF ADVERSARY-ORIENTED EVALUATION

APPROACHES AND THEIR CONTRIBUTIONS

OWEN [1971]

GUBA [1965] Suggested that educational evaluation might use aspects of legal paradigm

The first real adversary evaluation in education was conducted “modified judicial model”

WOLF [1975] “judicial evaluation model” (JEM)

10

THREE GENERAL APPROACHES TO ADVERSARY-ORIENTED EVALUATION

The Judicial Evaluation Model and other ‘Pro and Con” Adversary Hearing

Adversary Hearing with More than Two Opposing Views

Adversary Debates and Other Forensic Structures

11

THE CHARACTERISTICS OFThe Judicial Evaluation Model and other ‘Pro

and Con” Adversary Hearing1. Procedural rules must be flexible2. There are no strict rules for the assessment of evidence. 3. The only requirement is that the judge(s) must determine

before hand whether evidence is admissible or not.4. The parties may be asked before the hearing to present all

relevant facts, pieces of evidence and names of witnesses/experts to the judges

5. A copy of the complaint must, before the public hearing takes place, be committed to the judge(s) and the defence. The defence may plead guilty to some charges and deny others.

12

THE CHARACTERISTICS OFThe Judicial Evaluation Model and other ‘Pro

and Con” Adversary Hearing6. Witnesses are able to speak freely and may be subjected to

cross-examination.7. Experts may be summoned for a statement before or during

the hearing.8. Meetings of all parties involved with the judge(s) prior to the

public hearing tend to soften the debate and can be conducive to a joint striving to get to the truth of the matter on the basis of relevant facts.

9. Besides the two parties involved, other stakeholders may also be allowed to participate.

13

4 STAGES OF ADVERSARY-ORIENTED

EVALUATION

THE ISSUE SELECTION STAGE

THE PREPARATION OF ARGUMENTS STAGE

THE HEARING STAGE ITSELF

THE ISSUE GENERATION STAGE

Wolf (1979) and Thurston,propose the following four stages for adversary evaluation:

14

STEPS OFADVERSARY-ORIENTED EVALUATION

THE ISSUE GENERATION STAGE Should the program laid

off and pick some new programs suitable

alternative?

At this stage, a broad range of issues are identified. Thurston recommends that issues which reflect those perceived by a variety of persons involved in, or affected by the program in question, are taken under consideration in the

preliminary stages. 15

THE ISSUE SELECTION STAGE

This stage consists of issue-reduction. Wolf (1979) proposes that issues on which there is no debate, should be eliminated. Thurston states that this reduction may involve extensive analysis (inclusive of content, logic and

inference). The object of debate should also be defined and focused during this stage ( Wolf, 1979)


16

THE PREPARATION OF ARGUMENTS STAGE

PRO CON

This stage consists of data collection, locating relevant documents and synthesising available information. The data or evidence collected should be

relevant to the for and against arguments to be deployed in the hearing (Wolf, 1979)


17

THE HEARING STAGE ITSELF

This stage may also be referred to as the clarification forum and involves public presentation of the object of debate (Wolf, 1979). This is followed by

the presentation of evidence and panel or jury deliberation.


18

THE CHARACTERISTICS OF Adversary Hearings With More Than Two

Opposing View

1. Some of the review panels discussed in chapter 8 may hold public hearings to collect information pertinent to their charge

2. Appointed commission charged with the resolution of controversial issues frequently hold hearings to obtain evidence and opinions relevant to their mission.

19

3. House (1980) has cited as one example the frequent use in England of commission and councils headed by prominent citizens to provide guidance to government policymakers.

4. Several types of committee hearings are structured to identify and explore all the points of view represented in particular context.


Opposing View

20

5. Although not adversarial in the strict sense of world because:

Smith (1985) has noted they explore a variety of positions. prefer to include them because:

a. they reflect multiple viewpointsb. they frequently use hearing processes,

questioning, cross-examination, interaction concerning alternate viewpoints

and summary statements of the various positions.


Opposing View

21

6. St John (undated) referring to such hearings as the committee approach to evaluation:

All of those with a stake of evaluation A public hearing with testimony, questioning, cross-examination, and

summary statements The hearing method consists of public, verbal, face to face interaction. A process of communication and education occurs, and evaluation

makes its impact as it is happening.


Opposing View

22

1. Several approaches that qualify as adversary-oriented do not employ hearing processes.

2. Kourilsky (1973) proposed that pro and con arguments be presented to a decision maker. ( who would examine the evidence and question the presenter, ultimately arriving at the decision that seemed fair given both positions)

THE CHARACTERISTICS OF Adversary debates and other forensic

structure

23

3. Kourilsky and baker (1976) described an adversary model in which:

a. two teams prepared, respectively, affirmative and negative appraisals of that which was evaluated,

b. met to present the views to one another,c. Cross-examining and critiquing one another’s continents on

prespecified criteriad. Engaged in open ended discussions until reconciliation of

views was attained and translated into written recommendations in a single report


structure

24

4. Levine (1974) proposed that a resident adversary or critic might be assigned to a research project to challenge each bit of information collected, searching for other plausible explanations.

5. The Stake and Gjerde (1974) strategy of having two evaluators prepare separate reports summing up opposing positions for and against the program- yet another variant adversarial approach that does not depend on hearing format.


structure

25

6. Donmoyer (undated) proposed “deliberatively” approach to evaluation which focused on assessing and balancing alternative conceptions of reality and the differing value positions underlying these conceptions.

7. Nafziger and others (1977) described an adversary evaluation design employing a modified debate model of presenting data collected in a comprehensive evaluation to ensure that both sides of controversial issues were illuminated.


structure

26

Clyne ( 1982 ) summarized :-

27

THE STRENGTHS OF THE ADVERSARY- ORIENTED

EVALUATION APPROACH

28

1. Due to the public nature of the evaluation, openness and transparency regarding the object of evaluation is encouraged.

2. As the model takes into account multiple forms of data (inclusive of statistical fact, opinions, suppositions, values and perceptions), it is argued to do justice to the complex social reality which forms part of the evaluation.

3. The judicial nature of this approach may reduce political controversy surrounding an object of evaluation.

STRENGTHS OFADVERSARY-ORIENTED EVALUATION

29

4. As both sides of an argument are presented, the risks of tactical with holding of information should be minimized.

5. This approach allows for the incorporation of a multitude of perspectives, this should promote a more holistic evaluation.

6. The presentation of pro and con evidence and a platform which allows for cross-examination, permits public access to various interpretations of the evidence introduced into the evaluative context .


30


7. The presentation of rival hypotheses and explanations may enhance both quantitative and qualitative approaches.

8. All data must be presented in an understandable and logical way in order to persuade the jury. Dependent on the jury in question, this can make the data presented more accessible to the public and other stakeholders involved in the evaluation.

9. Finally, this approach is suitable for meta-evaluation and may be combined with other approaches which are participatory or expertise- oriented.

31

SUMMARY

The object of the evaluation affects many

peopleControversy

about the object of the

evaluation has created wide

interest

Decisions are summative

Evaluators are external

Clear issues are involved

32

THE LIMITATIONS OF THE ADVERSARY-

ORIENTED EVALUATION APPROACH

33

THE LIMITATIONS OFADVERSARY-ORIENTED EVALUATION

According to Smith (1985), many of the limitations of this approach relate to its competitive nature, the complexity of the process, and the need for skilled individuals willing to perform the various roles needed for a hearing. Listed are the main limitations of the adversary evaluation:

1.This form of evaluation may provoke venomous debate and conflict may have a negative impact on the outcome of the evaluation.2.The focus of the evaluation may shift to assigning blame or guilt, rather than optimizing policy.3.As adversary-advocate models are conflict-based, possibilities for reaching an agreeable outcome are curtailed. 34

THE LIMITATIONS OFADVERSARY-ORIENTED EVALUATION

4. Key stakeholders are not always equally skilled, and articulate individuals are placed at an advantage.

5. This method can be time-consuming and expensive (Owens, 1973)

6. It is sometimes difficult for hearing members to develop specific, operational recommendations (Wolf, 1979.

7. Time-limitations may only allow for a narrow focus.

35

THE APPLICATION OF THE ADVERSARY-

ORIENTED EVALUATION APPROACH

36

THE APPLICATION OFADVERSARY-ORIENTED EVALUATION

Although currently out of favour, this approach has been used quite extensively in the field of educational evaluation (Owens, 1973). It has also been applied to ethnographic research (Schensul, 1985) and the evaluation of state employment agencies (Braithwaite & Thompson, 1981).

Crabbe and Leroy contend that an adversary approach to evaluation should be beneficial when:1.The program being evaluated may affect a large group of people;2.when the issue in question is one of controversy and public attention;

37

3. when the parties involved realise and accept the power of a public trial;

4. when the object of evaluation is well-defined and amenable to polarised positions;

5. in contexts in which judges are likely to be perceived as neutral, and;

6. when there are sufficient time and monetary resources available for the method.

38

THE APPLICATION OFADVERSARY-ORIENTED EVALUATION

Adversary-Oriented Evaluation (2)

Documents

Transcript of Adversary-Oriented Evaluation (2)