EVAL 6000: Foundations of Evaluation

Dr. Chris L. S. Coryn & Carl D. WestineOctober 14, 2010

Stage 3 Theories

• Stage 3 theories represent an amalgamation of earlier theories (but still incomplete)– During this period theorists recognized the

need for contingency-based approaches to evaluation

– Emphasis was on specifying the conditions under which and for which purposes different practices are sensible

– Less emphasis on describing programs, more on explanation

– Recognition (mostly) that evaluation (mostly) can contribute to incremental changes

Stage 3 Theories

• Criterion 1: Theory of Social Programming– Stage 1 theorists operated on “naive”

assumptions (i.e., that radical changes would be made as a result of evaluation)

– Stage 2 theories and theorists recognized the “political” aspects of evaluation and directed efforts toward “incremental change” (Weiss, in her later work, reverted to more radical ideologies)

– Stage 3 theories recognized all of the same problems as Stage 1 and 2, but with an eye toward using evaluation as a means for making generalizations about programs

Stage 3 Theories

• Criterion 2: Theory of Use– In Stage 1, use was largely ignored (i.e., it

would occur naturally with truth)– In Stage 2, theories of use (by whom? For

what?) became the dominant concern of theories and theorists

– In Stage 3, while still emphasizing instrumental use, also promoted enlightenment as a means for summarizing what is known about a problem and potential solutions for policy making• Enlightenment was largely the domain of

experts (i.e., academics)

Stage 3 Theories

• Criterion 3: Theory of Knowledge Construction– Stage 1 theories prioritized truth, above all

else– Stage 2 theories from (just) truth to a

pragmatic position including knowledge that is both truthful and useful

– Stage 3 theories assert that no single paradigm for knowledge construction has sufficient empirical or theoretical support to dominate• Priority includes program description,

causation, explanation, and generalization• As with Stage 2, methodological pluralism

was advocated (though theorists of this period were critical of some methods of inquiry/discovery)

Stage 3 Theories

• Criterion 4: Theory of Valuing– Stage 1 theories included both prescriptive

and descriptive valuing– Stage 2 theories predominately advocated

descriptive valuing• Valuing is not a task for evaluators given

the complexities involved in valuing as well as the complexities involved in decisions about social programs

– Stage 3 theories advocated a “middle ground” that integrated both prescriptive and descriptive valuing and sources of values• Even so, single, summative statements of

value are still not seen as the task of evaluation given competing stakeholder interests and value positions

Stage 3 Theories

• Criterion 5: Theory of Practice– Stage 1 theories emphasized assessing

programs effectiveness at solving important social problems• Primary task is summative claims or

judgments • Eliminating bias was paramount

– Stage 2 theories emphasized methodological pluralism, although Wholey and Stake assigned central methods to practice

– Stage 3 emphasized contingencies under which certain methods would produce certain results• Still a strong emphasis on causal

knowledge and generalization• Also an emphasis on the mechanisms by

which programs produce results

Lee Cronbach & Peter Rossi

• Cronbach and Rossi advocated powerful theoretical positions (if accurate) that, to this day, have not been fully realized (and which are often misunderstood/misinterpreted)

• Independently and combined, their contributions represent radical departures from the positions and orientations of earlier theorists

• Although each drew extensively on the work of those who preceded them, both developed more extensive and sophisticated “theories” of program evaluation

Lee Cronbach

• Cronbach’s theory is complex, comprehensive, multifacteded, and more empirically-grounded than those before (and since)

• His theory is critical (and cynical) of social science theory, it methods, and how information derived through such methods should be used

• He seeks complex answers to complex problems, but with the intent of producing generalizable knowledge claims—disputed classic notions of validity

• Major interest was in individual differences (i.e., psychometrics)

Lee Cronbach

• Cronbach’s major contributions included, but are not limited to:– Measurement theory, in general, and

reliability, in particular– Generalizability theory– Generalization through explanation– Bandwidth and fidelity– Aptitude-treatment interactions– Generalizations from and to

• utos• UTOS• *UTOS• sub-utos

Peter Rossi

• Rossi’s theory emphasizes comprehensive and /or tailored evaluations with a focus on:– Program theories– Program life cycles or stages

• Eclectic integration or prior theorists work, but lacking detail about how and when to make decisions about tradeoffs (e.g., costs, time, expertise)

• Recognizes the legitimacy of different/equal values and needs– Emphasis is placed on needs as a primary

source of criteria/values

Peter Rossi

• Rossi’s major contributions included, but are not limited to:– Comprehensive, tailored, and theory-driven

evaluation– The “good enough” rule– The metallic and plastic laws of evaluation

• The “iron law”: The net impact of any program is 0

• The “stainless steel law”: The better the evaluation, the greater the likelihood that the net impact is 0

• The “copper law”: The more programs are designed to change individual, the more likely that the net impact is 0

• The “plastic law”: Only those programs that fail are likely to be evaluated

Further Discussion of the Readings

• What specific questions were raised as you read the assigned readings for this week (or past weeks)? Is there something specific that you didn’t understand?

• What problems remain unresolved regarding Stage 3 theories?

• What other issues emerged from the readings that need further exploration?

Role Playing as Major Theorists

• For next week– Each of you has been assigned the role of a

particular theorist (so, be certain to have a sound knowledge of your assigned theorist)

– In playing that role you will be asked to discuss the following from the point of view of your assigned theorist:

• What is the purpose of evaluation?• What is the role of the evaluator?• Is it the job of evaluators to make sure that

evaluations are used? If so, why and how?• What is the relevance, if any, of a theory of

social programming to evaluation?• What type of knowledge claims are central

to evaluation, and why?

Encyclopedia Entries for this Week

• Appreciative inquiry

• CIPP model• Connoisseurship• Deliberative

democratic evaluation

• Empowerment evaluation

• Goal-free evaluation