DOCUMENT RESUME

ED 226 067 TM 830 114

AUTHOR Gottfredson, Gary D.TITLE A Theory-Ridden Approach to Program Evaluation: A

Method for Researcher-Implementer Collaboration.Report No. 330.

INSTITUTION Johns Hopkins Univ., Baltimore, Md. Center for Social,Organization Of Schools.

SPONS AGENCY National- Inst. for Juvenile Justice and DelinquencyPrevention (Dept. of Justice/LEAA), Washington, D.C.;,National Inst. of Education (ED), Washington, DC.

PUB DATE Nov 82GRANT 80-JN-AX-0005; NIE-G-80-0113NOTE 27p.PUB TYPE Reports Research/Technical (143)

. .

EDRS,PRICE MF01/PCO2 Plus Postage... DESCRIPTORS *Delinquency; Educational Theories; *Evaluation

Methods; Instructional Improvement; National Surveys;*Program Development; *Proyram Evalliation; ProgramImplementation; Researchers; *Research Methodology

IDENTIFIERS *TheOry Based Evaluation

ABSTRACTSocial programs are often based on unclearly ,

articulated theoretical rationales. The evaluations of these pragrams

are also often relatively theory free, and therefore make limi4edcontributions to theory. A Program Development Evaluation (PDE)approach,, intended to integrate theory testing with the developmentand evaluation of action programs, is described and illustrated. Thisapproach has been applied for over 2 years in the national ev4uationof locally developed delinquency programs in 69 schools, and hAsresulted in (a) ,the implementation of several true experimentscpmducted in,collaboration with project implementers, (b) an icrease'in the clarity of projects' theoretical rationales, and (c) thq'identification and measurement of theoretical intermediary variables

as well as outcome variables. Based on the action research paradigm,the PDE approach appears to increase the theoretical and practitalrelevance of evaluation research. (Author/FN)

***********************************************************************Reproductions supplied by EDRS are the best that can be made

froin the original document.***********************************************************************

U S DEPARTMENT OF EDUCATIONNA J4,N.AI IWIT Ulf ;$ (WC AVION

teroaalanzaton

chothReport No..3.30November 1982A THEORY-RIDDENAPPROACH TO P OGRAM EVALUATION:A METHOD FOR RESEARCHER-IMPL MENTER COLLABORATIONGary D. Gottfredson

PERwsSioN to RERRODUCE THISMATERIAL HAS BEEN ORANTED BY

4-_

To THE EDUC,ATIONAL RESOURCESINFORMANON CENTER.(ERIC

".1

STAFF

J:dward L. McDill, Co-Director

lameg m McParcland, Co-Director

ri I. Alexander-

,

Henr, 1. Becker

10 ills H. Braddock, II

1-11titlev Brown

,Iti, 11. Carter

Michael Cook

Robert L. Crain

Dor:s R. Entwisle

Jov.ce L. Epstein

1A-es Fennessev,

Samuel A. Gordon

Deni:,e C. CotCfredson

Gar% '1). c'otttredson

Linda S. Cott1redson

L,b,ad 1. Harsch'

lohn H. Hollifield

Barbara I. Hucksoll

lots C. Hybl

kJ

Richard Joftk

Debbie Kalmus

Helene M. Kapinos

NanLy L. ,Karwesit

Hazel C. Kennedy

Marshall B. Leavey Or

Nancy A. Maddep

Kirk Nabors

Deborah K. Ogawa

Donald C. Rickert, Jr.

Laura Hersh Salganlk

Robert E. Slavin

lane St. John

Valerie Suncleyland

Cail F. Thomas

Villiam 1. Trent

James Trone

A THEORY-RIDDEN APPROACH TO PRCGRAM EVALUATION:

A METHOD FOR RESEARCH MPLEMENTER COLLABORATION

711

a.

Cary D. Gpttfredson

Report No. 330

Nalember 1982

This report is sponsored in part by Grant Is.b. 80 -JWAX-0005 from the,

National Institute for.Juvenile Justice and Delinquency Prevention, U.S.

Department of Justice, ahd in part by Grant Nb.,,NIEr-Gr-80-0113 from the

National Institute of Education, U.S. Department of Education. The

opinions expressed do not necessarily represent the position or policy

of either agency. Published ty the Center for Social Organization of

Schools, suppgrted in part as a research and development center by funds

from the United States National Institute of Education, Department of

Education.'

ter for Social Organizat,ion 9k Schools

,The Johns Hopkins University3505 Nbrth Charles StreetBaltinbre, Maryland 21218

Printed and asseMbled by the Centers for the Handicapped

Silver Spring, Maryland

The Center for Social Organizqpn of Schools

The CenteF for Social Organization of Schools has two primary dbjec-

'tives: to develop a scientific knowledge cf how schools affect their

students, and to use this knadledge to develop better school practices

and organization..

The Center works through,five programs to achieve its objectives. The

Steadies In 5chool Desegregation programapplies the basic theories cf

sodial organization of schools to study the internal conditions cf

desegregated Schools, the feasibility of alternative desegregation poli-

cies, and-the interrelations cf school desegregation 3.dth other equity

isgues such as housing and job desegregation. The Echool Organization

program is currently concerned with authority-control structures, task

structures, reward systems, and peer group,processes in schools. It has

produced a large-scale study cf the effects of open schools, has devel-

oped Student Team Learning instructional processes for teaching various

subjects in elementary and secondary schools, and has produced a compu-

terized system for school-wide attendance monitoring. The School Pro-

cesses And Career Development program is studying transitions from/high

school to post-secondary institutions and the role cf schooling in the

development of career plans and the actualization cf Labor market out-

comes. The _Studies in DelinquencY And School Environments program is

ekamining the role of school environments and experiences on-delin7

quency.

The Center also supports a Fellowships in Sducatioa Yesearch prograM

that provides oppottunities for talented young researchers to conduct

and publish significant resear.ch, and to endourage the-participation of

iqdmen and knorities in research on education.

This report, prepared by the Delinquency And School Environments pro- .

gram,. describes a method to make evaluation researdh more,theoretically

and practically useful.

Acknowledgements

An earlier,version of this report was presented at the Meeting of the

American Society of Criminology, Tbionto, NOvember 6, 1982. I am grate-

ful for advice from Deborah Daniels, Denise C. Gottfredson, John Holli-

field, and pne St. John on a draft of this report, and for discussions

with J. Douglas Grant and Carol Yamasaki.

A Theory-Ridden Approach to Program Evaluation:

7A Method for:Researcher-Implementer Collaboration

Abstract

Social prograns Arei often based on unarticulated or

-unclearly articulated theoretical rationales. Tne evaluations

'of these prcgrams are also often relatively theory free;-and

therefore make limited contributions to theory. A Program_

Development Evaluation (14x) approach, intended to integrate

theory testing with the development and evaluation cf action

programs, is described and illustrated. Tnis apprcech has.

been applied for over tao years in the national evaluation cf

locally developed delinquency programs in 69 schools, and has

resulted in (a) the implementation of'several true experiments

conducted in collaboration with project impleAnters, (b) anC"\

increalse in the clarity of,prcjects' thebretical rationales,

and, (c) the identification and measurement of theoretical

intermediary variables as well as outoame variables. Eased on

the action research paradigm, the FDE apprcech appears to

increase the theoretical ahd practidal relevance cf evaluation

research.

This report is about a critical

task_facing behavioral scien-,tists--the task of designingresearch that advances both theory

and practice. Educational andsocial programs are gten based onunabticulated or undlearly articu-

lated theciretical rationales. The

evaluations of these programs arealso often relatively theory free,and therefore make limdted contri-butions to theory.. A Colleaguesand I (Gottfredson, Note 1) have

created a Program DevelopnentEvaluation (PDE) method intended

to make possible the implementa-tion and testing cf stronger, moretheoretically guided action.pro,grams. Ws approach has teen

applied far ovdr two years in thenational eValuation cf locallydeveloped delinquebcy prevention

programs 69 schools, and hasresulted in (a) the implementationof several.true exIxriMents con-ducted in caboration with proj-ect implenenters, (b) an increase

in the clarity cf projects' thee,- 1

retical rationales, and (c) the_identification ancl neasurenent oftheoretical intermediary variablesas well as outcome variables.

Based on the action research para-digm, the FDE approach appears toincrease the theoretical and prac-

tical relevance cf.evaluationresearch.

The FDE method has evolved from ,

efforts to help school systekS andcommunfty-based organizations, createeffectj.ve programs and to evaluate

those pin:grans. A primary task

for behavioral scientists is thedevelopment of knowlege, butresearchers who have worked with ,

,

practitioners struggling to solve-problems know that the behavioralsciences have nore to offer thanthe techniques cf research,smosgasboard of previouslyiestedinterventions, and a handfdll of

*thbories: Knaaledge about'theways organizations behave and

.fr

Theory and Evaluation

about the psychology cf ccisiOnmaking and planning in groups canbe used to assist organizationsnot only taexamine the results oftheir efforts, but also toincrease organizational effective-ness in adopting theoreticallyplausibae innovations and assuring$eir faithful implementation

The tDE method provides astructure to merge the roles andactivities of organization develroper, theoretician, and evaluation.researcher. This method is a formcf actiOn research (Chein, Cook, &Harding, 1948; Lewin, 1947; San-

ford, 1970). Ihe approach assumesthat the prospects for promotingchange are greatest when organiza-tional decisionmakers' stake inthe research is made cleat by

their own participation in theresearch. DeciSion makers andresearchers collaborate through acontinuing dialogue in whichresearchers provide feedback onthe consequences cf project ,

action. Aotion researCh'involves,a cycle of hypothesis formulation_and planhing, action, evaluation'and inforthation feedback, and then

renewed hypothesisiformulation and. planning. AsAthe cycle id

repeated, and information derivedfrail project efforts'and researchis used in decision making, pFoj-ects Should bedome more effec-tiveturning the process into an'upward Spiral of'activity. And as

the cycle is repeated, theorYtesting is refined, resulting inbetter; more practical theory. ,

, Projects usUaliir change over

.tArne41 the basis of the expeii-:

ence gained as'they develop& Gottfredson, Note 2). Wnat

Pehrl (Note 3) has called "quality. , .

control" is needed to insure notjust that a program is run accord-ing,to thd plan, but that_a, plan

exists and is modified to coincidewith the way a project, as it

a

4

-2-

2

develops, is actually run. Many

attempts to demonstrate the effec-tiveness of specifiable socialprograms have failed in part -

bObause plausible interventionswere not implemented or' theirimplementation was not documented(Quay, 1977; Sechrest, White, &Brown, 1979; Ball & Loucks, 1977),or the plan for the innovation wasnot clearly articulated at the.outset (Sarason, 1974.

The PDS nethod facilitates the'study of the development ff a prc,gram by assisting in the planning

process. It provides a mechahism-by which an organization can makeits plans and-the theoretical ,

rationale behind them explicit,and then engage in theory-relevantresearch as it goes about imple-menting and studying its actiVi-ties. It also helps theresearcher monitor and documentproject implementation as theproject evolves. In shortothe.pLE method is an attempt to inte-grate theoretical research withevaluation and organizationaldevelopment. Its action'researchapproaCh to knoaledge generationAnd organizational,grcwth isaerived,from a tradition of con- *

cein for practical theory, usefulresearchr and organizationalchange and development.

-

Guiding. Principles

The evolution.and Practical tise'of the EDE rethod- is guided;by'seven'general principles. These .

srinciplessare used to,nake deci-siOns about the conduct of the ,

evaluation research and to xpsolveambiguities about appropriate nextsteps. Essentially, these guidingprinciples are a theory aboutdoing theory-relevanW researchwhile increasing organizationaleffectiveness. Evaluationresearch conducted in accordancewith these seven-principles,should

be most productive. Sinaarly,organizations that implement pro-grams using these principles

should be more effective thatorganizations that ignore them.

I. Actign projects guided by,explicit,theories that can betranslated intO practice will be

most effective. Not all theories

can be translated into action byany given project, and all theo-ries are not created equal. Sci-

ence progresses by creating ideasand eventually rejecting theunproductive ones; some ideas haveshown utility, and other thingsleing equal, ideas that have sur-vived previous empirical testsshould have considertble prece-dence. Uhe theory guiding a proj-ect should' be a template for it,

with decisions and,interventionsjudged against it at the same timethat the theory is judged by the

effectiveness of the interven-tionS. Projects will,he imple-mented with pc4 enthusiasm, be

_strongest, and contribute most toknowledge if (a) the ,theory isgenerated by or regarded as sensi-ble by the project implementersthemselves, and ib) tiw theoryatcdrds with evidence from previ-"oukresearch and evaluation.

,

2. Effective 4doption of an

jpnovatiomor other,interventionis more likely whtn explicit plansfor adoption_are available, andperceived obstacles to organia-tional change are seen as likelyto be overcome ty a °conceivable .

plan. ,

- 3., Effective implementatim of

an intervention or-innovation is' more likely if (a) blueprints for

(nanuals, proto-cols, etc0 are available, or if

. the interVention is structured by. forms, rules, or operating proce-

dures; and (b) implementation is,'subject to data guidance, i.e.,

Theory and Evaluation

observation and feedback of infor-mation to workers about the degreeto which their behavior aocordswith the behavior specified by theblueprints for the intervention.-Effective blueprints include plansfor data ,guidance, and provide fot

tdocumentation of the implementa-tion of interventions.

N, 4. PrOjects will intrease in\effectiveness under evaluation

pressure. This pressure takes

many forms, the most important ofwhich are: (a) pressure to focus

on theory in examining Organiza-tional behavior and the behavior

of the,organization's inhabitants;(b) pressure from potentially use-ful knowledge or information ofrelevance.to the organization; '

(c) pressure frocin "personal know-

ledge" based cn many sources,including direct observation orexperience; (d) pressure from the

rigorous, theory-based evaluatidn'of intervention components;(e) pressure.fram the rigorous,theory-based evaluation of proj-ects as a whole; (fr pressure fromfeedback about steps taken,to

'adopt an innovation; and .L

(g) pressure fran feedback aboutsteps taken to implement an inter-

vention. '

5. Projects internalizingthese principaes will behave inaccordance with them more often

than projects that sinpay complywith the application of them, andthe former will therefore ulti-mately be more effective.

6:--Theimore direct* projectimplementers benefit from evalua-

tion, the more evaluation will beintegrated with project opera-

tions.

7. The interests of prcjectimplementers and evaluators coin-

cide because'cn of the best)prays:

to create commun cable knowledge

Theory and Evaluation

is'through the rigorous, ehguided evaluation of well imple-mented interventions that Can bedescribed so well that others canunderstand What was done andtherefore replicate them. Both

implementers and reaearchers heedrigorous evaluation, the adoptionof innovations, welLimplementedinterventions, thorough descrip-

'tion, and theory./".

These guiding principles appearto us to accord with sound organi-zatiOn development practice(French & Bell, 1978), sound fieldresearch practices (Empey, 1980),and practical Wisdom in evaluationresearch (Tharp & Gallimore, Nbte

4). They were used to create the

PDE method.AO.

Organization Development sumAntecedent of=

One of the roots of ProgramDevelopment,Evaluation is thepractice of organization develop-,

ment (Cp). French,and,Bell (1978)

characterize OD as a pfocessinvolving actiOn research thatemphasizes normative change, isbased in behavioral science, 'involvea experience-based learningof intact work teams, and empha-sizes goals and objectives. By

characterizing op as a process,French and Bell mean that CD is 1

t"not to be regarded as aone-shot solution to organi7zational problems, but.moreas a 'growing toward'greater effectivenessthrough a series of intep-vention activities over?aperiod of tima . . .,Chang-ing the culture of . . an

entire,organizatiOn'is along-term, involved process"

(p..69).

In addition, they see CD as a pro-cess involving rational, empirical

strategies, but one that is even' #

more dependent On normative-reedu-cative strategies: ."The client

defines what changes and improve-iments he or she wants to make,rather,than the change agent; thecnange agent attempts to intervene

in a mutual, collaborative waywiththe client as rhey togetherdefine problems and seek solu-

tions; anything hindering effec-,tive problem solving is brought tolight and publicly examined" (pp.

75-76). The emphasis on normativeeducation is based on the assump-'tion that behaviors are rooted innorms, values, or beliefs as well

as in rationality'and Self-inter-est., CD is a data-based approachto planned change in which infor-mation is a spur to'action.Unpleasant information is not to

be avoided but rather treasuredbecause it.may lead to advance-ment, to clarification of prob-

lems. lypically, CD empilasizesconcrete goal setting through theshared experience of a group in

'formulating plans. Ihe on-the-joblearning experience of an intactgroup is presumed to promoteorganizational and individual

effectiveness.

The interactive, collaborative,participetive approach, often usedby behavioral scientists or CDspecialists 'serving as consultantsor facilitators of organizationalplanning and.decision making, hasmuch to offer in overcoming someof the difficulties a researcheffort may expect to face. First,

increasing an organization'seffectiveness Should increase thelikelihood that it will succeed inimplementing interventiohs with a'possibility of being Shown to be'effective when subjected to seri-ous summative evaluation. ',Second,in the CD process, the scientistapproac4es an organiiation in aManner that may decrease theextent to which he or she is

perceiyed as an alien invader. By

helping an organization clarifyits goals and Objectives, byassisting in creating open commu-nication about problems, and ryfostering the expectation thatprojects will change and developover time, the researcher may Come

to be considered More as an ,

insider, an entity to be trustedto convey useful news. And, theperspective that information, evendndomfortable information, itvaluable in fostering grcwth andconfronting important problems maydecgease the organization's usual

feat of evaluation. Finally, the

links between OD and actionresearch make the integjection of

formal research possible.

The Program revelopment Evalua-tion method'is in pert a descen-dent 4 an CD method previouslyused by the Social Action ResearchCenter (Blanton & Alley, Note 5)in a'series of projects to manageand study social change. This

predecessor, called the Program'Development (PD) model, was devel-

, oped through attempts to elealvate

i human service projects. In the PD

model, feedback is a mechanism ofptoject development that involvesmonitoring a project's environ=ment, the implementation of stra-tegies, and the achievement of

goals. in practice, the ProgramDevelopment specialist focuses oninteraction withqccject imple-menters to assist in assessingneeds', in articulating goals andmore specific objectives, in ana-

lyzing a project's forcefield(environmental constraints andresources), and in developing .

strategies for change or implemen-

tation. Like other forms cf OD,

PD emphasized participetory plan-ning in part to foster normativeN,,_reeducation and in part toincrease organizational and indi-vidual competencies in decisionmaking and planning.

Theory and/Dialuation

Action Besaargh Jan .an Antede,rit, a _HZ

FTE has itd roots in action

research. According to French andBell (1978), the.oridins of actionrnsearch lie in the work.of tewey(1933), Collier (1945), and Lewin(1946), The toots of actionreeearch are, however, deeper than

this. Tney can be traced back tothe Baconian formulation of thescientific method, which specified

three'steps: (a) the fOrmation ofhypotheses, (b) the empiricaltesting of the hypotheses, and(c) the acceptance or, rejection of

the hypotheses (Deese, 1972):,Action is takento "twist thelion's tail" ta learn aboutndture. Since Bacon, science has

been active rdthersthan specula-

tive, historical, or'reflective.Dewey translated the scientificMethod cf problem solving for lay-persons, and Collier and Lewinboth applied the scientific method .to solving practical social prob-

lems. ,

Collier (citedin French &,Bell, 1978), d commidsioner ofIndian Affairs concerned withimproving race relations, wrote ofaction research, claiming that:

,U

Rdsearch and then'moreresearch is essential to theprogram, that in the ethnicfield research can te made atool of action essential toall other tools, indeed thatit ought to be the master .

But we had in mind aparticular kind of research,

or, if, you will, particular

conditions. Vie had in,mind'

researth impelled from cen-tral areas cf needed action.Ard since action is bynature notonly specializedbut also integrative.. . .

our needed research must be

A

'Theory ind Evaluation

of'the integrative sort.Again, since the findings ofthe research must be criti-cized.by them through their '

experience, the administra-tor and the layman muse.'themselves participate crea-

tively in the research,impelled as 'it is froth their'

..town area Of need. ,(p. 94).

broader attention was called to

action research by Lewin (1947),

an eminent and influential psycho-

logical theorist'with a keeninterest in the applications'ofpsychology. .b1e saw that coopera-

tion between the change agent (Or

field worker) and the,researcheris important forboth planning and

mpagement:

Planning starts usuallywith something like a gen-eral idea. For one reasonor another it seems Qiesira-.

ble to reach a certain

objective. . . . The first

stelco. then, is to examine

the idea catefu4y in thelight of the means'availa-

ble. Frequently more fact-

finding about the situation

is required. If the first .

period of planning is suc-cessful, two items emerge:an 'overall plan' of.how toreach the objective and altbision in regarerto thefirst step of the action.Usually this planningalso somewhat modifiedoriginal Idea. The nextperiod is devotea to execut-ing the'first step of the

overall plan'. . . [and) by

certain fact-findings. . .

This . , . faCt-finding has

four functions. It should

evaluate the actionjoy.show-ing whether what has beenachiOved is above or belaa

ti

I.

-6-

4

expectation. It shouldserve as a basis for cor-rectly planning'the nextstep, {for} modifying the'overall plan.J Finally, it

gives the planners a Chanceto iearn, that is, to'gather

Pea general insight . .*

regarding the strength orweakness.of certain .

techniques of action. .

Rational social manage-ment,therefore, proceeds iha gpiral of steps each ofwhich/is composed of a cir-cle of planning, action, andfact-f indijig about the

result cWthe actiOn. (pp.

3337334). *

i

This sequential and spiralingmethod of problem solving is nowwidely used in organizationalaevelopment efforts, and has beenapplied in a variety of indus-trial, human service, and educa-

tional action research projects;end it appears to be at the heartof Tharp and Gallimore's (Nafe 4)Evaluation Stccession model.

Several varieties of activity ''

are often Called,action research(Chein, dook, & Harding, 1948).Sometimes the effort is limited todiagnosis and recommendations;sometimeS organizations Or projectimplementers carry out the entire

process; sometimes records crdiaries of actions taken and their-perCeived effects are maintained.Tharp and Gallimore (Note 4)describe several ways of "know-ing," each appropriate to differ-ent stages in'the development of aprogram... What they ball experi-

mentation,", "qualitative/personalknowing," "data guidance,"*.and"program evaluation" are all use-

Aul in program develOpment andresearch. But the varietY ofaction research most productive of

S.

trustworthi knbwledge is exEeri-

mental action research. Unfortu-.

nately,,experimental_actionresearch is-also the mdst diffi-

cult to perform, because it,reguires the conditions necessary

for confident inference (Cook &,

Campbell, 1979; Gottfredger Note1,Chap. 3) ,.and a statae aet ofinterventions that the organiza-

tion knowskhow to.and can imple-

ment in teStable form. Seldom do

CD efforts aim to implement "exper7

imental aCtion research,,largelybecause it is so difficult.Iñlementi4g experimental action

'research i however, a chief. aim

of the ..Pro4ram Development Evalua-

. tion method. Because of the pace

.of organizational change, rigorousexaniinatior4 of the consequenies of

actions may not always be timelyand less rigorous ways of knowing

about the effects of innovations

are often n4cessary. But research

mustibe coo dinate rafher thansubordinate to problem solving;solving problens without learning

how or why they were solved,will-contribute little to organiza-tional effectiveness or to theory

in thelong run.

.Tbk Meths/di

The FDE method emphasizes

. (a) theory, (b) measurement, and

(c) experimental or 4uasi-experi--mental design to a greater extent

than do many approaches to organi-

zational change and development.In addition, some cohmonly used

terms (most notably "objectives")

are defined in a sPecial way.

Mastery of the prabtical meaningsof the terms in the PDE structurewill provide change agents with a 941*

language for thlAking about,facilitating, kanaging, and study-

ing their change efforts.,

The Program Development Evalua-tion method/ illustrated in Figure

incorporUes theory as an

'I., Theory and Evaluation

explicit,component, give6 nuggia-12.1e goals and objectives.i hard- .

nosed meaning, and incorporatesplanning for evaluation implemen-tation in the same way that plan-ning for-any other aspect of a ,

project is inborporated. It

alloWs project Implementeis andevaluators to monitorcriticalbenchmarksin'the adoEtion of\anystrategy to createCtiange, and it .

allows them to monitor mmplemehta-tion Standards in the implementa-

tion of interyentions. The,prin-J

cipal concepts involved in the FDE

structure are elaborated below:

Troblers and-Goals

'Work in any project shouldbegin with an expaoration cf its

intent. Most organizations, and

most researchers, have-multipleaims. Within the EDE framework, ,amoverarching aim is called a

goal. A goal is the obverse of aproblem; it specifies.how thelevel of the problem may be meas-ured and therefore how one mayknow if progress is being made.

r4ary questioni arediscussing goals.ion.terves to '

and enable evaIu-

Several secoimportant whThe first quesreduce ambiguiation; it asks how each goal may

. be measured. The second question

serves to proNte realisticresearch designs; it asks when aproject can realisti4ily expectto make a substantieft_differenceand therefore specifieb the dura-tion cf intervention and the tim-

ing cf measurement. And the third

question, essential in tnaerimen-tal or qpasi-experimentai actionresearch, asks'how one,may kncw

that the project itself was -

responsible tor prdiress towar.ds-

the goal. These questions are, of

course, steps tward involvingproject implementers in the'designof the research. . And, they serve

to make explicit what the grgani-iation-expects to accompaish.

.

cif.

i

-1,' 4*

---- ....._

ure 1: The.Program Development Evaluation Method

I t----- ---__ _ :

,

,

%

. ....,-7:----30--MeasurableProlAems , Goals

,

Feedback.SystemData .Gatherin9. Inference

t-

Benchmarks,Standards, Tasks

I

.,

s

,

1li

,Theory of

ction

..

. .

5. ForceFielct:Analysii

Nitil -,Ei

r

...

cgSr ,

MeasurableObjectives.

Choice ofInterventions

*,

,

Theory and Evaluation

nex.1

Actions.are taken for reasons

that are either articulatedr.br

umarticulated. Ihe PDE methOd is

a vehicle for making theory,expli-

cit., This is useful because; as

the Panel on Research on Rehabili-

,tative Techniques Martin,Sechrest, & Redner, 1981) notes:

In attempting to solve anyproblem, a clear idea of the

nature of the problem, itscauses, and developmentalprocesses ,is vital. In the

absence of.an adequate con-

ceptual framework . the

ruSh cf enthusiasm-for an .interesting intervention islikely to short-circuit con-sideration of these factors.

The result is . . . efforts

that mpf-be unrelated to thecauses of,crime, ignore the

most suitabae target popula,-tions, andtail to considerquestions of optimal tindng

and strength of,the inter-vention. .The adoption of atheoretical framework ndees-

sarily prcupts considerationof the above factors and,one hopes, thoughtful devel-

opment and implementation cf

; . interventions, therebyfriereasing. ti_la chances

effecti.veness.. trf:

Theory helps to organize know-

ledge and to_commenicate,'it pr5-

vides a guide for action, and it

assists in developing and assess-

ing interventions.

"Once a basic problem isstated in theoretical terms,plamers have an explicitfoundation on which to build

an intervention strategy and

from which to derive aresearch strategy in con-junction with the

-9-

1

intervention" (Martin etal., 1981, p. 3.4; cf.

Glaser, 1980).

In short, an explicit theory Pro-

vides a template for projectirplementers' use in buildingtheir interventions, as well as a

1:gmp1ate by which both implement-

ets and researchers can assess

those interventions. Therefore,

t4e PDE process calls for deliber-

ate and careful Consideration cf

the question, "Why do these prob-lems exist?" When an intervention

is designed using theory, its

evaluation tests the theory under-

girding the intervention.

Ojectives

In the language of PDE, anobjective is an intermediary out-come that a project's theory.of

action implies it important. Li*wals, objectives must be stated

4Ik measurable terms.

Some examples may help make thee

distinction between goals andobjectives clear. Wppose that a

change agent wishes to'decreasethe-death rate due to gastroenter-itis in a rural society. The

change agent theorizes that thesuffering and death are due to thecontamination of village witef,supp1ia7with-the dholera.microor-.ganism. This theory might suggest

a campaign to chlorinate wells,

with the Objective Of decreasingthis contamination. Me objectivewould be measured by laboratoryanalyseS of well-water sanpaes'todetermine the levels of microbialcontamination,,and attainment of

the goal might be Measured by...counts of deaths per 100,000 popu-lation due to gastroenteritis.Another dhange agent Atight see the

problem somewhat,differenbly.This second change agent maytheorize that the suffering anddeath are due to poor

Theory and Evaluation'

environmental sanitation: Because

few villagers use sanitarylatrines, well water is easilycontaminated and the choleramicroorganism spreads frominfected to uninfected persons.This theory might suggest an envi-

ronmental sanitation campaigndirected at persuading villagersto 'construct sanitary latrines and

sanitary wells. The objective nowinvolves villager behavior, andTight be measured by the propor-tion,of households using sanitarylatrines and water from protected

wells: A theory can, of course,suggest multiple intervftions andmultiple objectives. lhe second

change agent's theory would alsoreasonablWmply chlorination of'wefls and,assesbments of wellwater. 'The more cOmprehensive atheory, alemore complex the arrayof interventions and objectives itis likely to.suggest.

Change agents could developtheories at many levels to explain

the problem of dholera deaths, and.each level would suggest somewhat'different interAtions. lb con-

tinue-the ekatples, change agents,might attribute the problem to(a) normative beliefs in village

--societies-that_current standards.of environmental s-iiiMtiOn.are 'adequate, (b) the poverty and ,segregation cf the rural people,which depiive them of theresources to build sanitary

devices and concentrates them sothat they are' at high risk,

(c) social stratification thatallaes Only an elite merchant .

_class access to sufficientresourres to enjoy a sanitaryenvironment, (d) stratification inthe wvrld system that enablescapitalist countries tolkeep coun-

tries with rural rubber-tappingpopulations impoverished and thecost of raw materials law. Each

of these theoiies may have consid-

erable validity. Yet each would

imply different interventions tosave the problem, ranging fromdumping chlorine in wells ba over-throwing the capitalist world.sys-

tem. NO single cholera preventionproject is likely to attemptinterventions at all of these lev-els, and so will not have objed-tives at each level. A prOject'stheory ofactionthe theory ,Oatdrives its interventionsib ithetheory that is relevant in specif-ying objectives.

Again, answers to several keyquestions--how objectives nay bemeasured, when effects are,to beexpected, and how one may knawthat the intervention caubed-theeffectsr-serve to create the eval-uation or research design. In,addition, the explicit statementof objectives serves to make clearto the organization what itexpects to haPpen as a result of

its efforts.

Intervention

An intervention iS an action '

taken bo achieve'an objective or"

set of objectives. Ordinarily, itis a major component of a project.The _term is often synonymous Oith"change,"-Itreatment," Of "compo-

nent." Some'interventions areaimed at changing the behaviOr,attitudes, or status of individualpeople; others are aimed at chang-ing the behavior of an organiza-.

tion or collectiVity. An inter-vention is a process, action,structure, rule, or substance thatis apaied or put in place to,--achieve an Objective or set ofobjectives, and therefore bo move'clober tg achieving a goal. An .

intervention may be chemical,

physical, biological, behavioral,social? political, pr structural.The interventions employed shbuldbe aimed at the objectives anOrganization's ttleory of'actionimplies mhst be achieved:

r 1 I

-

Forcefield

A forcefield is the soCial-psy-chological field that immediatelysurrounds a decision or action.It includes the forces that compelor restrain against alternativeactions as they are perceived byan individual or .corporate egtor.The notion of a forcefield.com4sfrom Lewin's (1951) ideas aboutthe field of forces influencingaction. An examination or analy-sis of an organization's force-field, especially 'one that focuses

on the field-in terms of.theresources available and the Obsta-cles to action, is frequently use-ful for four reasons: (a) By

focusing on the organization'sperceptions of environmentalinfluences, the nature of these.perceptions becomes explicit andopen to,scrutiny, revision, amen-dationr'supplementation, and test.

(b) A complete account of obsta-Cies and resourcff decreases the.likelihood that either pitfalls orpotentials will be overlooked inthe development bba project.(c) Using knowledge of the influ-

ences in the prolect's environmenthelps to capitalize on opportuni-ties or arrangements that gobeyond the'resources imder a ptoj-ect's direct control. (d) Alter-native strategies or plans toimplement any intervention can toecreated and assessed in the oon-:

text of the forcefield. Careful--

attention to the forcefield sur-rounding a-research projectincieases the likelihood that bothinterventions and research designs

will be implemented as intended.

Because initial analyses cf aforcefield may te 'objectivelyincorrect, because perceptions 'change over time, and because theaction of a projeCt may alter itsforcefield, the dynamic,nature ofthe field is to be exPeOted. Asensible practice, therfore, is to

Theory and Evaluation

renew forcefield analysis pericai-

cally, especially when any stra-tegy being executed on the basisof an initial forcefield analysisis not working well. Breakdownsin experimental design or in theirplementation cf an intervention

would both required renewed analy-sis.

, For more discussion of force-

field analysis and examples of itsapplibation see Hersey and Blanch-

ard (1982, pp. 115-119, 269-272).Practical guidance ori working withan organization to.enalyze its,forcefield is provided by Blanton& Alley (Note 5, 103-113)..

Strategies.

Strategies are plans.. ,p.pf:ord-

ing to the EDE rrethod, straegiesare developed from a forcefielaanalysis, just as objectives andinterilentions derive fram a 'theory

of action about a problem. Sev-

,eral possible strategies forimplementing a research project brone,cf its component interventionsare likely to exist. The bask forproject implementers and those whoare attempting to facilitate stra-.tegy development is to create a'

plan that is perceived as feasibleand attractive: If a criticalpath.in,some plan is blocked andno way around the cbgtacle is per-ceived, the plan is not a goodone. Alternativesaths thatobjedtively exist but have notbeen perceived will not be fol-lowed. (This point illustrates

why thorough and treative force-field analysis is helpful.)Strategy that appears workablewill make use of an organization'sresources to overcome the obsta-cles to itplementation and .

research. Such a strategy mayinvolve (a) moving around an'obstacle, (3) decreasing thestrength cf the forces workingagainst ixii.ementation, (c) turn:-

Theory and Evaluation

,ing,an obstacle into a resource,oril(d) using a strategy in which

4 ,the obstacle is irrelevant and.leed.not-be overcome.

A fully articulated strategy istscomposed of three kinds ofele-

`ments: critical benchmarks,implementation standards, andtasks.

C111.10,1 benchmarks. A criti-cal benchmark is a key decision,agreement, action, or arrangementnecessary to move forward with aplan. A benchmark is much like agate that must be opened to mOVealong a path. If the gate doesnot open, progress in executingthe strategy is blocked. The

locations of these benchmarks (orthe nature of them) are made clearin the process of analyzing theforcefield around an intervention.For example, the forcefield analy-sis about a project's efforts toprovide in-service/training forteachers might imply thab anobstacle lies in teacher unwill-ingness to participate in trainingoutsiderof normal working hours, ".

and that a resource is the author-ity of the deputy superintendentof schools to grant release tineand to allocate the funds fo'r sub-stitute instructors, The deputy

superintendent's.agreementtogrant release time and to author-ize the expenditure for substitute.teachers would then Ipecome a,cri-

tical benchmark. Ihe deputysuperintendent is a gaiekeeper(Lewin,41947, p. 333) whose psY-chology must.be examined to learn

. how to get'the gate opened.

Specifying when a critical'benchmark is to be acComplishedfaciliChtes Management. Any stra-,tegy will require a temporal or.

.logical sequence of milestones/that must bemet. In the forego-ing example; a failure to accom-plish the critical benchmark would

signal the need to devise a'newstrategy for getting ttledone, or the need to seek analternative to training.

Imp1ementation:Btandar4s. Thesecond part of a strategy for theimplementation of both interven-tion and research are a set ofimplementation standardq. Sound

research practice has longinvolved attention to surveyresponse rates, the measurement ofindependent and *pendent varia-bles, and the procedures used toinduce experimental manipulations. Researchers in experimen-tal social psycholOgy and physiol-ogy routinely conduct manipulationchecks to determine that implemen-tation standards have been met.In research on the effects cf die-tary supplementation or deficien-'cies for example, investigators donot assume that an artificial diet

has been composed ag intended butperform quantitative enalyse onsamples.cf the actual food tcj det-

ermine that the diet conne was. as:intended (e.g., VOhra, Gott-.fred8on,,A gratzer, And,

. careful researchers on labelinfeffects perform manipulation .

checks to see that interpersonalexpectanCies hwe been experimen-tallY modified as intended (e.g.,Eden4 Shard, 1982).

vImplementatiOn,standards in a

prevention research project might

inclOde the specification oE such'intervention Characterlstics .as(a) the skills, knowledge, andnumbers of staff; ib) the fre7quefty,'duration, and content ofinteractionsof workers withclients, families, dr communityorganizations; (c) the specificactions to be taken in a range.cfspecified situations; and ,-

(d) gUidelines for the naturesandvalue of reinforcers to be appliedfor specific kinds of performance.In general, inplementation

standards should be sufficientlycondete and specific that Oey ,

allow for a comparison-of-what isbeing done with what is intended.

Tasks. The third part of a

strategy is the set etasksrequired to execute it. A task,statement specifies who will do

,36/hat,b Iten. Specifying a person

to be responiible for executing aparticula; task, even when a groupwill be involved, promotes clar-ity. And specifying when a task'is expected to be completed is anadditional management tool.

\

. ,

Critical benchmarks, implemen-tation standards, and tasks allserve important functions in proj-ect management and worker rein-

forcement: They serve to guide an

organization's effortd, and theyprovide one kind of objectivestandard of achievement. A lack

of such objective standards"deprives the workers . , . of

their legitimate desire for satit-faction on a realistic basis.nder these circumptances, satii-

(:faction or dissatisfaction with

tone'sLown-athievement becomesmainly 'a question of temperament"(Lewin, 1946, t4 35).

Development4

, At the veryiheart of the PDEmethodlis the expectation thatproject development and research

. will be *an_ongoing trocess!_and'. ....

'that the projeale- environment,is' dynamic. Only an'effete organiza-

tion is immobile, at equilibrium.411bTe sion, reastessmenti.review,

1416 wiping, and phanges in actions .

t,akeh are the hallmarks of vigo-

. rolls projects. Consequently, EDE

is a cyclica.1 process cf action

research at progress is madetowards achieving goals and objec-tives (or as goals and objectivesare redefined), as new informationbecomes available, as the

Theory and Evaluation

environment changes, and as new,

research questionsemerge.

Development occurs largely:,

thrmgh the use of information.Information about the achievement

soL nonachieyement of critical

benchmarks signals that the force-field has been,usefully under-stdod, or that developmental,effore is required to reassess theorganization's forcefield. Infor-

mation that an objective is being,achieved signals that an interven-

tion is effective( and informationthat an objective is not beingachieved signals a reconsideration

of the appropriateness, strength,or fidelity.of the intervention,and_prompts new planning. Infor-

mation that there is progreSstowards a goal signals that the

organization is OD the righttrack. Information that there isno progress towards the goal maypignal several things, dependingon the.pattern,of other,feedback.If interventions are being imple-mented as intended 'and Ehey areachieving their objectives, thetheory is called into question.If objectives are not.being met,either the theory or integrity ofOeintervention, or both, ShoOldbe sciainized. Success in bring-ing about elusive objectives andsolving seriout problems is not to.

be expected at once. Bgt the PDE

structure is intended to provideinterim feedback on,progress to

enable a strengthening of.theXoject. ,Evidence of projecteffectiveness lends support to thetheory guiding the project.

Evaluation

The PDE structure is intendeito facilitate several kinds of

evaluation. The explication of atheory of action allows an assess-ment of its plausibiliEy, and anassessment of the plausibility orstrength of the project's planned...

-13-

Theory and Evaluation

interventions in light of the

theory.

Ey tracking the achievetent of,critical benchmarks, the structureallows assessment of progresstowards implementing an interven-

tion. Monitoring implementation.staniards provides for the assess-

ment of '04)401 2nterventions arebeing executed --it provides mani-pulation checks for both research-

ers and project managers. Tnese

are key elements of formativeevaluation and the conduct,of

research. ,

The.PDE method is also intendedto facilitate rigorous summativeevaluation--it promotes experiMen-tal action resedrch 1Chein, Cook,& Harding, 1948) or at least qua-si-experimental action research.It repeatedly,asks the question,

"How do we know the interventionmade the difference?" The imple-

mentatidn of an evaluation deSign,is treated in the-same Way ad'theimplementation of any other inter-

vention. The PDE method assumes ,

that evaluation is an essential

L. -component of effective projectdevelopment and should receive ,

'Opordinate effort-with Other:-

aspects of 'project implementation.Therefore, forcefieid analysis isperformed for design and data col-lection issues just as it is fot

any other project component.Because project implementers areinvolved in the researchAesign.and in the specification gf the.research questions, their commit=

ment to strong evaluation i$expected to,increase. And,

beCause the forcefield analysis10cuses on the project implement-,ers' own forcefield--their Percep-tions of the possible--the immedi-.ate environment of the evaluationis taken into account when theevaluation,is designed, Perhapsmitigating.some of the resistance,

to evaluation activities corilmonlyJ. 'I

encountered among implementers.

Strength And Fidelity. .,

The PDE method makes possiblethe assessment of the"Strength andfidelity of planned interventionSthrough judgments about theoreti-'cal plausibility and benchmark

monitoring. This assessment can

occur ih two ways. First,.projectimplementers can assess the con7

sistency oftheir interventionsand objectives with the theory ofaction underlying their project.That is, a project implementer candetermine whether theobjectivessought accord with the theory,,andwhether the interventions plannedwill plausibly achieve the proj-ect's objectives. .In shoit,

_theory is a template for makingjudgments abOut the appropriate-

ness of interventions and dbjec-

tives that project iholementers ,

can use.to quality control their.

own projects. Second, observets'

of a project, including research-ers, can assess its prioristrength by determining whetherthe planned interventions willplausibly lead to the objectives,.or goalsof the projectiby_assess-,

. ing them in comparison to state-.of-the art theories in the field'

-"and the history of similar proj-ects that have been conducted in

'the past.

As a project develops decisions

to adopt an innovation will havebeen tade,,and the nature Of theevents important to the project

changes. EarlY in a project'sdevelopment, major issuesyillhave to do with ideas for inter-ventions or With strategies for

mLgetting an innovation adopted. At

-later points, important issueswill involve the integrity of theintervention's implementation andthe assessment of effectiveness.In thi early stages of,movementtoward'the adoption of an

,

innovation, critical benchmarkswill involve events related to the

decision to adopt a change.Later, 'implementation standards or'quality control checks on thefaithfulness cf the implementatidnwill increase in importance.

The documentation and assess=

ment of interventions as imple-

mented may involve detailed manu-als for the administration of,treatments or programs; descrip-±ions of the characteristics cfstaff and target groups; andaccounts of the duration and sche-duling of,treatments or events,treatment protocols, or proportion

of the population served. Mere-fore, as projects develop, PDE,increasingly focuses on the devel-

opment of manuals to guide service

delivery, make diagnoses, andtrain the staff. Action research

,projects using this model willdeVelop strategies to monitorstaff perforipance, provide incen-

, tive structures to keep perfor-

, mance aCcording to specifications,

and the like.."The implementationof those strategies is,expected tohave two consequence's: (a) 'the

plans and their execution.wouldinctease the integrityof the . _

:intervention, and (b) the infOrma7----tkon-generated by,the imPlementa-

tion cf these plans would describe

the integrity of.interventions.Manuals developed in the course ofcarefullyQimplementing a project

.will.allow for subsequent closereplications in future research or

application..

, ,,swe Exam lea

Two actual examples of projects

nod unde4Way provide insight into

the Way the POE method enables thetranslation of practitioner ideasinto theoretical terms and pro-

notes project develOpment. First,

consider an exanple of a clin-Ouncy prevention project

\

*Theory and Evaluation

involving a, peer group interven-

tion. Basically, the projectinplementers assume that delin7quent behavior,is supported by a

normative subculture. To prevent

delinquency the project intendedto cpmpose groups of delinquent

and non-delinquent youths and to

st,ucture group interaction suci

that delinquent behavior andexpressions of beliefs counter to

traditional moral beliefs are con-

fronted in the group. Altering

beliefs is an impo%ant objective.This project's theoreticalrational and statement cf inter-vention standards evolved, over a

period cf over two yearein which

, a field worker for the evaluation

(Jane St. John) and project imple-

menters (Sonny Luster and William

Kottman) used the ZOE nethod.

Early research results'for this*project implied that theoreticalintermediary objectives were not

always being.met. In early

results A measure of belief inconventional Social rules wasnegatively effected ty the treat-

ment for non-delinquent group par-ticipants. Uhis research outcome

wab used by the project implement-ers to revise their trfatment spe-

cifications and to deviseways to

monitor group processmbreclosely. Specifically, group

leaders were provided additionaltraining, more attention was given

Eo the composition of the groups,

and monitoring of grail) ihtefac-

tion using the Hill 11977; Note 6)interaction matrix. was initiated.

Although the development and eVal-uation of this project is stillunderway, the early negativeresult has dibappeared in sUbse- 7

quent evaluations.

A second actual example is al

project currently being conductedby a Southerp urban school system.Essentially/ this project's theory

assumes that assistance and pres-

sure to improve academic

-15-

204:

Theory and Evaluation

perforbance, 'crabined with the

opportunity to deNcelop attachnentdto-pcosocial others, will result

in more rewarding experiences inschool, greater attachment, andless truancy and delinquent behav-

ior. This theory suggests thatstudents experiencing academicdifficulties and who are disrup-tive in school should receive spe-cial attention from project per-sonnel in the form cf tutoring andadvocacy on behalf of the studentwith teachers, parents, and oth-ers. The evaluation,field worker(Denise Gottfredson) and project

staff (Doris Coaxum, Barbara Ell-ligard, Ann Birdseye, and MarthaStuart) psed the PDE'method. .

Early implementation nonitoringshoWed that the intervention wabnot being implemented as intended,with project personnel interactingmástly with students who did notfall within the, target group, andinteracting only, to a lindted

extent with target'students.

Clearer implementation tandards

were specified for the workers tofollcm; monitoring of thesestan-dards increased, and subsequent

-- research resultsshowIimprovedimplementation and have begun to

shcw increased,academic performance as measured both by stan-dardized test scores and schoolgrades, and some evidence of'decreased school .clisruptivenesssand misConduct when randomly equi-valent treatment and control stu-

dents are compared. Someintermediary outcomes, notablyattachment to school, wre not .1

affected as expected, suggestingthat the program can Ite strength-

ened further.

Limitations znd PotentialCriticisms

The PDE.methdd has same limita-tibps, creates some tensions, and

is open to.criticism. The most

'f*

inportant-limitations appear to be

that it.is complek, it is time andexpertise intensive, it fails to

completely resolve the eensionsspmmativelevaluation causes forproject imPlementers, it is aninTerfect mechanism for coping .

with the separate goals of projectsponsors and implementing organi-zaXions when these are not com-pletely in.accord, and it con-

fronts researchers and ,.

implementers with tough decisionsinvolving the sacrifice of rigor-ous research designs in 'order to

achieve some aspede of project

implementation.

.0.0 Talent

The human behavior required tosuccessfully implement the EDEmethod' is complex, and the meth-od's implementation calls for aninvestment of human resources.Use of the EDE structure calls for"high levels of interpersonal com-petency, tact, patience, communi-cation skill, and understanding:of'group relations in organizations.'

In addition,'it calls for=arough undergtanding of evaluation 5methods=-measurement, social sci-

ence theory, experimental and qua-si-experimental methods, and

statistics. Ironically, this com-bination of competencies are

. rarely found in one and the sameperson, suggesting that a team of

workers may be required,to-imple7ment this approach to organiza-tional change. Furthermore, the

cyclical or developmental naturd'of PDE requires constant (or at'least frequent periodic) atten-tion, monitoring, updating, andinformation _cornmunitation. This

taxing process goes beyond theeffort typically expended in

research.

, Some trade-offs are likely to

be involv s fing an action

iesearch ffort using the EDE

method. Because staff with.therequisite skills are hard to find ,

in a ty ical organization, train-

;ing wil be required. pcperience

implies h at social scientists_trained primarily in researchmethodology, statistics, andtheory can successfully impl nt,

the method, but that thel reqtgieadditional training in organiza-tion development to do so:

Tensions

Tension appears,endemic in

eva1uat4In research. 'Roo often in

the pastb evaluatiOn ha's teen used

as a tool for canoeling a proj-

ect--even when positive evalua-tions could not reasonably beexpected at an early stage of

project development. tension is

also created by the inherentlypolitical environment of actionprojects, and ty environmentswhere the successful project doesnot rock the boat. the develop-

ment of sound programs usuallyrequires the expenditure of tineand moneyand Often imrlies thenecessity of arrangements that aredisruptive.

Tradeoffs-and Research _Rigor

Program 'Developmeht Evaluation

is value laden. Participation ofproject implementers is a fupda-Mental principle in the PDE pro,

cess; pursdit.cf the goals and

objectives of the implementingorganization are generally assumedto be desirable (although open to

question). Furthermore, an.ainick

PDE is to develop the implementingorganization's capacity to accom-

plish its.goals. 'therefore,

researchers and impleMenters col-laborate in evaluation design, ID

question formulation, and plan-ning: As a result, researchersextensively intervene in projectdevelopmentindeed they become a ,part cf the pcoiect.

Theory and Evaluation

Some evaluators (Perloff, 1979)'

see this as undesirable in,a sum-

mative evaluation because itraC§es questions about the gener-alizability,cf the results tosituations where researchers are

absent. . In addition, just as

research heeds sometimes intrudein 'project operations byxreatingnew tasks or structural arrange-

merits, the pursuit of a project'sprogrammatic activities can result

in compromises in research design.

As Deutseh (1968, p. 466) says,"the,danger that confronts theresearch worker in such situations

is the possibility that hisresearch design or methodology ,

will...be sacrificed to the achieve-

ment cf the social-action objec7tive.".

A. 'This "danger!' nay account in

part for the reluctance academicsocial scientiss have shown topartiáipate in action research.This danger seems a smell price to .

pay in exchange.for the opportu-

nity_action research.creates to -

contribute tothe-solution' ofsocial'problems, although thetradeoffs involving evaluationrigor e&e painful.. In short,'the

PEE, metbod is no panacea for this

tough prbblem.'

Complexity

. The PEE method is complex.Unfortunately, each Component of

the method seems at present to be '

useful and'desirable in action,reserch efforts. nevertheless,

this complexity suggests that aMore streamlined Method is appro-

priate when doing,short-termorganizati9nal development inter-ventions that do not have a ,

research,purpcee. ConseguentlY,

for many btief organization devel-orment interventions, the selec-'tive use of those portions of thePDE structure that seem to be themost relevant for the probleth at

-17- 2 2.

o

Theory and Evaluation

hand is more appropriate thanattempting to use all parts of the

method. Researchers Should not

abandon any part of the nethod.

'Practical Application

Experience implies,that one oranother part of this process isuseful to_projeCt implementers;indefining their van jobs, inrformu-

lating plans, and in clarifying

their intentions. Experience also

implies that the entire pro ecss issametimes viewed as burdensome.On'the whole, this structure seemsacleat_improvement over saMe more

traditional research methodibecause it involves impaementersin research planning, because itexplicitly attempts to build sum-mative evaluation structures based

on an organization's fabefield,and because,it focuses on goalsand objectives of cancern to

ingleMenters.

MY colleagues and I are using

the PDE method in the theory-rid-den evaluation of delinquencyprojects conducted by 17 different

organizations. In an area where,true experiments are almost unk-nown Mixon & Wright, 1575; Gott-fredson,:1981; Ogawa, Nbte.7J, sixof these projects are implementingtrue experiments. Our opinion is

that the quasi-experimentaldesigns of most of the remainingprojects are sUbstantially betterthan would have been the case hadproject impleMenters not beeninvolved in.an evaluation usingthd PDE method. Even "where theoriginal researCh design has bro--

ken dawn, project implehentershave usually Shared:with the

.researchers the task of creatingalternative designs and in devel-

oping plans to prevent subsequentbreakdowns.

The greatest virtues'of the PDEmethod appear to be (a) its

ability to make Possible theoreti-

cal tests,through research chaction projects, '0:04.ts ability

to elaborate clear, measurableintermediary outcomes, or-cpjec- .

.tives, useful'in assessing the,efiectivenes$ of interventions in-theoretical terms, (c) itS ability

to provide project implemepters -

with the tools to assess their awnefforts by neasuring interventionsagainst theory and objectives,(d)its ability to generate creatiVe strategies petceived as feas-ible to implementers based on thedivergent and then conVer4entthinking that takes'place'inforcefield analysis, (e) 4ts abil-

ity to ,involVe project implement-.

ers in the retearch enterpase,byengaging them in the specification

..44*neasurable goals and objec-tives, and in the creation ofevaluation designs, (f) its abil-

ity to provide Short=term assets- .

ments.of progress throUgh themonitoring,of critical benchmarks,implementation standards, and _

tasks, (g) its ability to enableresearchers to understand thenature of a prcject ty translatingimeAementees.ideas into a struc-

turedlanguage.of theoreticaltresearch, and (h) its ability to

selve ad a structure for communil-cation between researchers and

practitioners.:

PrE makes serious'organiza-tional change and rigorousresearch more attractive, to organ-izations despitedts inability ta-make it'trUly'paiatable to all of

them. Ideally, praCtice and .

research would be merged into asingle enterprise in which rigor-ous theoretical tesearch becomesah integral component of program

operation. It'is unrealistic to,

thinkthat most practitioners willevdr acquire allCf the technicalskills required to Systematically'conduct rigorous research on theiractivities (just as init.

-18- 2 3

unrealistic to expect mostresearchers to become adroit prac--1

titioners). In addition, truly, rigorous research is not aways_called for in the, development cf aproject; and not all action proj -

.ects test theoretically interest-ing ideas. When rigorous evalua-

,

ifk

Theory arki EI;aluation

tion is called for, however,',the .

PDE structure is helpful. ProgramDevelopment Evaluation does notsuccessfully resolve many Of tliesourcesAof tension in mergingaction with research, but it is

'progress.

24:.-19-

I.

Theory and Evaluation 4

Reference Notes

1. Gottfredson, Q. D. The ephool action effectiveness study: First

Interim report (Report No. 325). Baltimore: The Johns Hopkins Univ-

ersity, Center for Social Organization of Schools, 1982.

2. Wilkins, L. T., & Gottfreasonl D. M. Research, demonstration, and

aodal Loam'. Lvis, Calif,: National Council on Crime-and Delin-

quency Research Center, 196.9:

3. 1,)arl, A. Ouality control In evaluative repeatch correctional

_A'et, programs. Paper presented at the4Nationa2 Institute on Crime apd

DelinquencyrJuly 1962.

4: Tharp, R. 2, & Gallimore, R. Thd ecology Dragitram research ma-

development: ki =del at evaluation succession-. Unpublished manu-

script. (Available from Roland Tharp,.Department of Psychology,

University. of Hawaii, Honolulu 96822; or Roland Gallimore, Department

of Psychiatry and Biobehavioral Sciences, UCLA, Westwood, California

'90024.).

5. Blanton, J., & Alley S. IlAgama.deyeldpment: =awl Is& Dzaniza-

tional seif-stOy. (Prepared under oontract.no HSM-42-72-143 froat

NIMH).' Nicasio, Calif: Social Action Research,Center, 1975.

6. Hill, W..F. Hill interaction matrix {Rev..ed.). .Los Angeles: Univ:

ersity of Southern California, Youth StUdy Center, 1965. 0,

7. Ogawa, D. K. Preventing delinquency: The record of accomplishmerit.

In G. D. Gottfredson (Ed.) jadagol =Lim effectiveness 4udy: First

intgam report (Report No. 3255. Baltimore: Johns Hopkins Univer-

sity, Center far Social Organization of Schools, 1982.,

4.

-2o- 2,6

N

4( Theory ahd Evaluation'.

References3,

.

Chein, I., CoOk, S., &'%rding, J. Ille field'cf action research. Amer,.

iCan, Psych61ogipt, 1948, 1, 43-50. ,

, Cook, T. D.., %, Campbell, D..f. Ouasi-experimerptione. Esidn And =AD-

ysis issues for field get4ngs. Chicago: Rand McNally, 1919.

Deese, J. Psychology as science and Art. ttgf York: Harcourtirace

JOvanovich, 197Z.'. -

.,

.Deutsch, M. Field theory in sobial psychology. In G. tindzey% E.

Aronson (Eds.), Mg bandbQok DI sociaIpsy0bglegy .(Vol. 1/ 2nd'ed.) .,

Reading, Mass.: Addison-Wesley, 1968. ,

)Dewey, J. Pad ssm think (Rev. ed.). New York: Heath, 1913-.

Dixon, M. C., & Wight, W. E. Juvenile aelinquenty prevention programs:

An evaluationA policy zeigtedLefiesucth Da the, effectiveness s& pre-4,

vention nrograms. Washington, D.C.': National Science Foundation,

1975: .

-.

,

Eden, D.,.& Shani, A. B. Iygmalion goeito boot camp: ExPectancy.

.leadership, and trainee performance. loarnal at ilm1itil2sychol9car.;1982, 04 1.94-199.

r

Empey.i L. T. Field experimentation in criminal justice: Rationale and

design. In M.,W.,Klein & K. S. leilmann (Eds.), Handbook De criminal

justice.evalua'tion. Beverly Hills, Sage, 1980.

French, W. L., & Belli C. H., Jr. Organization development: pehavigral

science interventions Is= organization improverent (2nd ed.). Engle-

wood Cliffs, N.J.: Itentice-Hall,"1978.

dlaser, D. 7he interplay of theory, issues, Icaicy, and4data. In M. W.

Klein & K. E4 Teilmann (Eds..), Handbook DI gr4manal justice evalua-

'1,1th.* Beverly Hills: Sage, 1980. .

Gottfredsonl G. D. Schooling and delinquency.. In S. IL martin, 'L.- 'B.

Sechrest, ic R. Redner (Eds.), New directions j414.12.e' rehabilitations&

gLiminal offenders. Washington, D.C.: National Academy Press, 1981.,

Hall, G. E.,'& Loucksio S. F. A; developmental model for determining

whether the treatment id actually implemented. Ikericat EducatiOnil

itesearchsigarnal, 197,7. IC 263-276:

Hersey,'., & Blanchard, K. Management Joi, organizational behavior (4th'

E ed,). Englewood Cliffs, N.J.:. Prentice=Hall, 1982.,

mom,. W.T. Hill interactiOn matrix (HIM):, The conceptual framework,

derived rating scales; and an updated'bibliography. anall Gratip.

Behavior, 1917,41, 41-268.

Theory add' Evaluation

Lewin, K. '

Issues;

'Lewin, K.

Hartley1947.

Lewin K.

Action research and minority prcblerns. Journal Alt Social.

1946, 2, 34-46. ,

Group decisionrand iocial- change. Tn T. M. .2t.twcorob & E. L.

(Eds.) , Readings in social psychology. Dew York: Holt,

Field 'i:heory in sobial science: tew York: Harper, 1951.

MartihrS. E., Sechrest, L. Bo, & Redner, R (Eds:)." nog directions inthe zehapilitation's& grimjnal. offehdecs ORqport cf thekPanel On'.

Research on Rehabilitative TechniquesY. Washington, D.C.: National

Academy Press, 1981.

Perloff, R. (Ed.). Evaluator inventigns: 4,1ros and cons. *Beverly

Hills, Calif.: Sage, 1979.

Quay, 11. Three faces of evaluation: What can be expected to work.

Criminal Justice,end,Behavior; 1977; AJ 341.1-354.

Sarasod, S. B. Ile culttire of the school and the problem .of change.

Boston: Allyn & Bacon, 1971. .

Sanford, N. itatever happened to action research? ilszunai Di SocialIssbes, 1970, 2.6., 3-23. .

Sechrest, L., Whiter/S. O., & Brawn/. E. B. tids.). Me rehabilitation

_of grimiL4 offenders; Problems And prospects (Report of the Panel

on Research on Rehabilitative Technitlues), Washington4 D.C.:

-National Academy of Sciences, 1979.

,libhra, P. Gottfredson, G. D., & Kratzer, F. H. The'effects of high

levels of dietary EDTA, zinc or copper on the mineral contents of

some tissue of'turkey poults. Poultry Science, 1968, AL 1334-1343.

It