Getting to Maybe: Evaluation History Systems...

21st Annual RTC ConferencePresented in Tampa, February 2008

Michael Quinn PattonTampa 2008

Getting to Maybe:Evaluation,

Systems Thinking,and

Complexity Science

Tampa February 25, 2008


Evaluation History

Setting the Context:Context Matters

So, In the beginning…

Michael Quinn PattonDecember 2007


Utilization-FocusedEvaluation,

4th edition, May, 2008

•1st edition,1978•2nd edition, 1986

•3rd edition, 1997


New Direction # 1

International and cross-cultural expansion ofevaluation:globalization and diversity


New Direction # 2

From Studies to Streams



Evaluation Trends:20 years ago

• One study for one user• Modest databases• Long time frames for studies• Presumption of direct use• Long reports• Generally single method• Dissemination was the written word


10 years ago• One study for multiple users.

The age of stakeholders• Larger databases with computer support• More client focus• Not just decision use, but also conceptual use• Multiple teams producing information• Quantitative/qualitative wars come to an end:

valuing multiple methods


And now…• We are moving from

discrete studies toinformation streams

• Systems not individual evaluatorsproduce evaluative knowledge

• Evaluative streams are multiple –integrating information from differentsources


And now…• Data collection & reporting at multiple

levels by multiple-stakeholders• Databases are continuous and virtual• Time frames are immediate• Analysis is continuous• Virtual analysis of trends and conditions• Visual displays instead of narratives


And now• Partnerships are dominant in collecting,

analyzing and sharing evaluativeknowledge

• Internet is the new information glue• Increased transparency of evaluative

knowledge• Emphasis on continuous organizational

adaptation and improvement


New Direction # 3

Proliferation of evaluation models,theories, options, and methods,and approaches



Original Primary Options

Formativeand

SummativeEvaluation

(Mid-term and End-of-Project Reviews)

Blandin CommunityLeadership Program

DevelopmentalEvaluation

Evidence-based Practice

Evaluation grew up in the “projects”testing models under a theory ofchange that pilot testing would lead toproven models that could bedisseminated and taken to scale:

The search for best practicesand evidenced-based practices

Fundamental Issue:How the World Is Changed

Top-down dissemination of“proven models”

versusBottoms-up adaptive management


Conditions that challengeevaluation

• High innovation• Development• High uncertainty• Dynamic• Emergent• Systems change


Challenge:Matching the evaluationprocess and design to thenature of the situation:

Contingency-basedEvaluation



New Direction # 4

Broader understandingand

conceptualization ofevaluation use


Conceptualizing Use• Utilization-focused evaluation

now includes knowledgemanagement, organizationallearning, and facilitating change.The focus is as much oninstitutional uses of knowledgeas on individual users


Re-conceptualizing Use

• Use is a process not a event• Use involves an interaction not just

a report• Use involves training for use not

just delivery of results• Use begins at the beginning not at

the endMichael Quinn Patton

Tampa 2008

New Direction # 5

Increased up-front role forevaluation & evaluatorsin intervention design:

Logic modeling &Theory of change work


Some premises:• Evaluation is part of initial program design,

including conceptualizing the theory of change• Evaluator’s role is to help users clarify their

purpose, hoped-for results, and change model.• Evaluators can/should offer conceptual and

methodological options.• Evaluators can help by questioning assumptions.• Evaluators can play a key role in facilitating

evaluative thinking all along the way..• Designs can be emergent and flexible.


New Direction # 6

Beyond linear logic models:

Systems Thinkingand

Complexity Science



Three ways of conceptualizingand mapping theories of change

Linear Newtonian causality Interdependent systems

relationships Complex nonlinear dynamics


Linear Logic ModelINPUTS (people, materials)ACTIVITIES (processes)

OUTPUTS OUTCOMES

CHANGES IN PEOPLES LIVES IMPACTS

CHANGES IN COMMUNITIES


Pushing Force(Non-directional)

Dissatisfaction withthe

Status Quo

(Inertia)a.k.a. “Cost of

Change”

Resistance toChange

Pulling Force(Directional)

a.k.a. “Desirability ofthe end state”

Compelling Vision


Pushing Force(Non-directional)

Dissatisfaction withthe

Status Quo

(Inertia)a.k.a. “Cost of

Change”

Resistance toChange

Pulling Force(Directional)

a.k.a. “Desirability ofthe end state”

Compelling Vision

First StepsBelievability

Causal Diagram of Beckhard’sChange Formula


Output / Product

Essential Attributes

Attributes required to meet of exceed

customer needs:

"Do the Right Thing"

Efficacy

Appropriate

Characteristics to meet or exceed

customer wants and expectations

of excellence

"Do the Right Thing Well":

Efficiency

Dignity and Respect

Effectiveness

Timeliness

Reduce Waste

Safety

Continuity

Availability

What inputs need to go into the

process to make the product

that produces the desired

result?

What steps need to be taken to

create the product that achieves

the desired result?

What features / characteristics should the product

have?

Systems Logic Model

Customer

Outcomes

&

Satisfaction

! Measure

Effectiveness

! Measure

Satisfaction

! Inform

Improvement

needs

Effect

Inputs

Staff Resources

Financial resources

Internal Standards

External Requirements

and Information

Equipment/Materials

Key Processes & Functions

Inputs organized and utilized

Procedures

Steps

Key processes

Measure Variability

Assess Process Control

Assess fidelity to planned

procedures

Assess impact of variation

Evaluate opportunity to raise the

bar

Cause

Feedback into process

What is the desired result?

What should customer

experience?

Planning

Implementation

Structure Process OUTCOMES

Feedback


Division workplace

that :

• Offers a healthy

work environment

• Recognizes

excellence

• Provides quality

training and

management

• Includes effective

systems ,

procedures , and

communication

( Goal 5 )

Increased adoption ,

reach , implementation ,

and sustainability of

recommended public

health strategies to

achieve strategic plan goals :

• Prevent risk factors for

heart disease and stroke (Goal 1 )

• Increase detection

and treatment of risk

factors (Goal 2)

• Increase early

identification and

treatment of heart attacks and strokes

( Goal 3 )

• Prevent recurring cardiovascular events

( Goal 4 )

Division for Heart Disease and Stroke Prevention Evaluation Planning Logic Model

Internal

Workforce that is :

• Diverse

• Skilled

Resources that are :

• Available

• Timely

Division Leadership

that provides

sufficient :

• Infrastructure

• Policies

• Strategic Planning

Leadership

Disparities

Surveillance

Research

Evaluation

Program

Translation

and dissemination

of the current

knowledge base ,

and identification

of ways to improve

that knowledge

base

Effective :

• Management

• Coordination

• Staff

development

Enhanced

competency of public

health workforce

Enhanced ability

of programs to apply

findings to improve

public health

Enhanced external

application of Division

goals and strategies

Increased advocacy

and “activated

constituency”

Engaged network of

states and partners

Enhanced integration

among chronic

disease programs

Increased focus

on heart disease

and stroke

prevention efforts

by states and

partners ,

especially with

regard to

disparities

Policy

Increased

knowledge of

signs and

symptoms

Improved

emergency

response

Improved quality

of care

Reduced risk

factors

Reduced

economic impact

of heart disease

and stroke

Eliminated

preventable

strokes and risks

Reduced levels of

disparities in heart

disease and stroke

Reduced

morbidity and

mortality of heart

disease and stroke

External

Planning ActivitiesTranslation ,

DisseminationAdoption , Practice , Sustainability Impact

WHAT WHYHOW

Communication

Collaboration



©TOM Jochen Enterprises, Möckernstr.78 10965 Berlin


Step Two: THEN A MIRACLE OCCURS“I think you should be more explicit here in step two.”


Systems

• Parts are interdependent such that achange in one part changes all parts

• The whole is greater than the sum ofthe parts

• Focus on interconnected relationships• Systems are made up of sub-systems

and function within larger systems


Systems Concepts in Evaluation –An Expert Anthology. 2006.Bob Williams and Iraj Imam

AEA Monograph,EdgePress/AEA Point Reyes CA.



Understanding theElephant

froma Systems Perspective



The relationship between whatgoes in and what comes out

Whatconceptualframeworkinformsfront-endevaluationwork?


Teen Pregnancy ProgramExample


Logic Model for Pregnant Teens Program1. Program reaches out to pregnant teens

2. Pregnant teens enter and attend the program (participation)

3. Teens learn prenatal nutrition and self-care (increased knowledge)

4. Teens develop commitment to take care of themselvesand their babies (attitude change)

5. Teens adopt healthy behaviors: no smoking, no drinking,attend prenatal clinic, eat properly (behavior change)

6. Teens have healthy babies (desired outcome)


Systems web showing possible influencelinkages to a pregnant teenager

Teachers/otheradults

Youngpregnantwoman's

attitudes &behaviors

Herparents &

other familymembers

Child'sfather &

peers

Prenatal program

staff

Her peer group


Program systems web showing possible institutionalinfluences affecting pregnant teenagers:

SCHOOL SYSTEM

Youngpregnantwomen's

attitudes &behaviors

PrenatalClinic andHospitalOutreach

Church

Prenatal program

Othercommunity-based youth

programs

Other Systems-- welfare-- legal-- nutrition programs-- transportation-- child protection-- media messagesContext factors-- politics-- economic incentives-- social norms-- culture-- music

YouthCulture


Sustainable change: Systems dynamic reinforcing feedback loops

Teenagerfeelsbetter

Teenagertells clinicnurseabout herchangedbehaviors

Teen tellsherfriendsand familyabout herchangedbehaviors

Teenagergetspositivefeedbackfrom clinicnurses

Teen getssupportfromfriendsand family

Teenagereatsproperly &takes careof herself

Pregnant teenagerlearns theimportance of notsmoking anddrinking -- andstops doing both.

ASustainableReinforcing

System



Using Different System Lenses toUnderstand a “particular” System

Biologic System• Emergence• Coordination/synergy• Structure, Process, Pattern• Vitality

Sociologic System• Relationships• Conversations• Interdependence• Loose-tight coupling• Meaning/sense

Mechanical / Physical System• Flow• Temporal Sequencing• Spatial Proximities• Logistics• Information

Economic System Inputs/Outputs Cost/Waste/Value/Benefits Customers/Suppliers

Political System• Power• Governance• Citizenship• Equity

AnthropologicSystem• Values• Culture/Milieu

Information System•Access•Speed•Fidelity/utility•Privacy/security•Storage

Psychological System•Organizing•Forces Field•Ecological/BehaviourSettings

SYSTEMDIMENSIONS


5

Map Systems as Webs

Source: Digital Capital: Harnessing the Power of Business W ebs,By Don Tapscott , David Ticoll and Alex Lowy


Complex Dynamic SystemsConfiguration


HIV/AIDS Example• Hits every system: health, family,

social, religious, economic, political,community, international

• Requires multiple interventions onmultiple fronts in all subsystemssimultaneously

• Resulting reactions, interactions,consequences dynamic, unpredictable,emergent, and ever changing


Some system premises

Systems neutrality:An observed system isfunctioning as observed forsome reasons, fulfilling somefunctions.In whose interests is a systemfunctioning? Who benefits?


Healthy systemIn a well-functioningsystem, no subsystem isoperating at its maximum.



Systems changeDuring transitions

from one system toanother, things will getworse before they getbetter.


Systems DynamicsDynamic system

interrelationships increase thelikelihood of unintendedconsequences as systemschange. Expect theunexpected.


Challenges:

SituationRecognition

andAppropriate

EvaluationDesigns


The nature ofEXPERTISE:

Situation Recognition


Contingency-basedEvaluation

• Situational analysis & responsiveness• Context sensitivity• Clarify and focus on intended users:

stakeholder analysis• Clarify and focus on intended uses• Methodological appropriateness• Criteria for evaluating the evaluation:

credibility, meaningfulness


Conceptual Options

•Simple

•Complicated

•Complex



Types of Community IssuesThe Stacey Matrix

Certainty

Agr

eem

ent

Close to Far from

Far f

rom

Clo

se to


Simple

Certainty

Agr

eem

ent

Close to Far from

Far f

rom

Clo

se to Simple

Plan, control


Following a Recipe A Rocket to the Moon Raising a Child

Complicated Complex The recipe is essential Recipes are tested toassure replicability oflater efforts No particular expertise;knowing how to cookincreases success Recipes producestandard products Certainty of same resultsevery time

Simple


Technically Complicated

Certainty

Agr

eem

ent

Close to Far from

Far f

rom

Clo

se to Simple

Plan, controlTechnically Complicated

Experiment, coordinate expertise


Following a Recipe A Rocket to the Moon Raising a Child• Formulae are

critical andnecessary

• Sending one rocketincreasesassurance that nextwill be ok

• High level ofexpertise in manyspecialized fields +coordination

• Rockets similar incritical ways

• High degree ofcertainty ofoutcome

Complicated Complex

The recipe is essential

Recipes are tested toassure replicability oflater efforts

No particularexpertise; knowing howto cook increasessuccess

Recipes producestandard products

Certainty of sameresults every time

Simple


Socially Complicated

Certainty

Agr

eem

ent

Close to Far from

Far f

rom

Clo

se to Simple

Plan, controlTechnically Complicated Experiment, coordinate expertise

SociallyComplicated Build relationships,create commonground



Socially complicated

Implementing human rightsagreements, like gender equity oroutlawing child labor

Environmental Initiatives Many different and competing

stakeholders Diverse vested interests High stakes


Socially complicatedsituations

pose the challengeof coordinating and

integratingmany players


Stakeholder MappingHigh Interest/ Low Power

THE INVOLVED

High Interest/ High Power

THE PLAYERS

THE CROWDLow interest/ Low Power

CONTEXTSETTERSLow Interest/ High Power


Know When Your Challenges Are In theZone of Complexity

Certainty

Agr

eem

ent

Close to Far from

Far f

rom

Clo

se to Simple

Plan, control

Zone ofComplexity

Technically Complicated Experiment, coordinate expertise

SociallyComplicated Buildrelationships,create commonground

Systems ThinkingRelationship BuildingCollaborationGood Enough VisionChunking Around DriversMinimum SpecificationsMultiple ActionsAdaptability & Organic


Following a Recipe A Rocket to the Moon Raising a Child

Sending one rocketincreasesassurance that nextwill be ok

High level of expertisein many specializedfields +coordination

Rockets similar incritical ways

High degree ofcertainty ofoutcome

• Formulae have onlya limited application

• Raising one childgives no assuranceof success with thenext

• Expertise can helpbut is not sufficient;relationships arekey

• Every child isunique

• Uncertainty ofoutcome remains

Complicated Complex






Simple


Know When Your Challenges Are In theZone of Complexity

Certainty

Agr

eem

ent

Close to Far from

Far f

rom

Clo

se to Simple

Plan, control

Zone ofComplexity

Technically Complicated Experiment, coordinate expertise

SociallyComplicated Buildrelationships,create commonground

ChaosMassive Avoidance



Following a Recipe A Rocket to the Moon Raising a Child• Formulae are critical

and necessary

• Sending one rocketincreases assurancethat next will be ok

• High level ofexpertise in manyspecialized fields +coordination

• Separate into partsand then coordinate

• Rockets similar incritical ways

• High degree ofcertainty of outcome

• Formulae have only alimited application

• Raising one childgives no assurance ofsuccess with the next

• Expertise can helpbut is not sufficient;relationships arekey

• Can’t separate partsfrom the whole

• Every child is unique

• Uncertainty ofoutcome remains

Complicated

Complex




Recipe notes thequantity and nature of“parts” needed



Simple


The Frogtown NeighborhoodChildren’s Community Initiative

in Saint Paul, Minnesota


Seeing Through A Complexity Lens

“You don't see something until you have the right metaphor to let you perceive it”. Thomas Kuhn Michael Quinn Patton

Tampa 2008

Complex Nonlinear Dynamics• Nonlinear: Small actions can have large

reactions. “The Butterfly WingsMetaphor”

• Emergent: Self-organizing, Attractors• Dynamic: Interactions within, between,

and among subsystems and partswithin systems can volatile, changing

• Getting to Maybe: Uncertainty,unpredictable, uncontrollable

Major Sources ofUncertainty

• Human irrationality:Behavioral Economics

• Different contexts• Change in all its splendid

manifestationsMichael Quinn Patton

Tampa 2008

New Direction # 7

Methodological Flexibility &Creativityversus

Methodological Rigidity



The Debate AboutRandomized Controls in

Evaluation:

The Gold StandardQuestion


Design Tension

Single StandardHierarchy

vsSituational Variation and

Appropriateness


GOLD STANDARD:

METHODOLOGICALAPPROPRIATENESS

not

Methodologicalorthodoxy or rigidity


Creative Design Thinking

David P. Billington:*

“The goal of good design is tointegrate efficiency, economyand elegance in a single design.”

* August 18, 2007, NY Times, A13


Photo byLynsey Gornick


“One Bridge Doesn’t Fit All”

“As many have pointed out, the deadlybridge failure in Minneapolis wassymptomatic of a system of bridges thatwill continue to corrode, crack and crumbleif not maintained. But maintenance is notthe only problem. We also need to designand build better bridges.”

Metaphor for evaluation design



“The Minneapolis collapse ishauntingly similar to the collapse in1983 of another interstate highwaybridge over the Mianus River inConnecticut. That disaster led toinspections of similar bridges, whichfound dangerous cracks fromdeferred maintenance….”


What the Mianus and Minneapolisbridges had in common was not justneglect. Both were the products of adesign mentality in which engineerssimply used a standard form, andoften the same detailed features.Public bridges are all too oftendesigned by anonymous teams, andthe results can be seen on ourhighways.


Creative Challenge

Situational adaptability: Contingency-based evaluation Appropriateness

--Using standard forms of evaluationand

-- Going beyond standard forms whenappropriate and useful


Standard Evaluation Forms1. Inadequate upfront utilization focus2. Program/project as the unit of analysis3. Linear logic models4. Focus on findings use vs whole process5. Individual outcomes focus vs systems

change6. Preference for quantitative data & RCTs

as the methodological Gold Standard7. Static designs


Getting to Maybe:How the World IsChanged? 2006Frances Westley, BrendaZimmerman, Michael Q. PattonRandom House Canada,


Complex Situations• Highly emergent (difficult to

plan and predict)• Highly dynamic, rapidly

changing• Relationships are non-linear &

interdependent rather thansimple (linear cause-effect)



Complex Interdependencies


Insert action into thesystem


EMERGENCE


Contingency-basedDevelopmental

Evaluation


Improvementversus

Development


BeyondjustSummativeandFormative



BeyondStatic Accountability

Models


Example of an emergent option:

DevelopmentalEvaluation


DEVELOPMENTAL EVALUATIONDEFINED

Evaluation processes, including asking evaluativequestions and applying evaluation logic, to supportprogram, product, staff and/or organizationaldevelopment. The evaluator is part of a team whosemembers collaborate to conceptualize, design and test newapproaches in a long-term, on-going process of continuousimprovement, adaptation and intentional change. Theevaluator's primary function in the team is to elucidateteam discussions with evaluative questions, data and logic,and facilitate data-based decision-makingin the developmental process.


CONTRASTS

Traditionalevaluations…Testing models

Complexity-based,DevelopmentalEvaluation…

• Supportinginnovation andadaptation


TraditionalEvaluation…

• Renderdefinitivejudgments ofsuccess orfailure

DevelopmentalEvaluation…

• Provide feedback,generate learnings,support directionor affirm changesin direction in realtime


TraditionalEvaluation…

• Measuresuccessagainstpredeterminedgoals

Developmental Evaluation…

• Develop newmeasures andmonitoringmechanisms asgoals emerge &evolve



Traditional DevelopmentalEvaluation… Evaluation…

• Evaluatorexternal,independent,objective

• Evaluator part of ateam, a facilitatorand learning coachbringing evaluativethinking to the table,supportive of theorganization’s goals



• Evaluatordetermines thedesign based on theevaluator’sperspective aboutwhat is important.The evaluatorcontrols theevaluation.

• Evaluatorcollaborates withthose engaged inthe change effort todesign anevaluation processthat matchesphilosophically andorganizationally.



• Design theevaluation based onlinear cause-effectlogic models

• Design theevaluation tocapture systemdynamics,interdependencies,

and emergentinterconnections



• Aim to producegeneralizablefindings acrosstime & space

.

• Aim to producecontext-specificunderstandingsthat informongoing

innovation



• Accountabilityfocused on anddirected to externalauthorities andfunders.

• Accountabilitycentered on theinnovators’ deepsense offundamental valuesand commitments –and learning.



• Accountability tocontrol and locateblame for failures

• Learning to respondto lack of controland stay in touchwith what’sunfolding

• And therebyrespondstrategically




• Evaluation often acompliance functiondelegated down inthe organization

• Evaluation aleadership function:

Reality-testing,results-focused,learning-oriented

leadership



• Evaluationengendersfear of failure.

• Evaluation supportshunger for learning.


Conditions

• High innovation• Development• High uncertainty• Dynamic• Emergent• Systems change


SenseMaker software• Dave Snowden, Founder of Cognitive

Edge, former Director of KnowledgeManagement at IBM

• SenseMaker can code and map 95,000stories in 24 hours

• See the world as others see it; anti-terrorapplications.

• See the quantitative patterns in the meta-data with qualitative context and meaning


New Direction # 8

Infusing evaluative thinking as aprimary type of evaluationprocess use.

Capacity-building as an evaluation focus.


Thinking about:

The role can evaluation playwith complex dynamicinnovations….




And the beat goes on…

Evaluation as anever-evolving field


References

• Getting to Maybe: How the World IsChanged? Frances Westley, BrendaZimmerman, Michael Q. PattonRandom House Canada, 2006.

• Utilization-Focused Evaluation, 4th ed.,Michael Quinn Patton, Sage Publications,2008

Getting to Maybe: Evaluation History Systems...

Documents

Transcript of Getting to Maybe: Evaluation History Systems...