AComplex-SystemsOrientation
toEvaluationthatSupportsa
CultureofHealth
Welcome!
Thisconceptpaperisaboutthesimplicityontheothersideofcomplexity.Itmaynotbeenoughtogiveyourlifefor,buthopefullyit’senoughtoinspireyoutothinkanddialogueabouthowyouframeyourapproachtoevaluationandtoconsiderwhatreframingmightbehelpful.
OnewayevaluatorsmaycontributemoreusefullytocreatingaCultureofHealthisbyunderstandingfundamentaldistinctionsbetweenthehierarchicalandnetworkedstructuresofsystemsandhowtheyareintertwinedincomplexsystems.Distinguishingbetweenthesetwostructureshelpsevaluatorsgroundtheirevaluationsinthefundamentalstructuresthatareofimportanceataparticulartimeandplaceastheysupportevaluationusersintheirintentiontotransformsystemstoservethepublicgood.
Everynowandthena[person’s]mindisstretchedbyanewideaorsensation,andnevershrinksbacktoitsformerdimensions.—OliverWendellHolmes,Sr.
Iwouldnotgiveafigforthesimplicitythissideofcomplexity,butIwouldgivemylifeforthesimplicityontheothersideofcomplexity.—OliverWendellHolmes,Sr.
vThePaper’sOrientation 2
vThePaper’sPurpose3
vKeyPointsofthePaper4 vAComplex-Systems
OrientationtoEvaluation27
vEvaluationGroundedinHierarchicalStructures29
vEvaluationGroundedinNetworkedStructures30
vContrastofEvaluativeFocusinHierarchical&NetworkedStructures33
v Culture35vACultureofHealth36vACultureofHealthActionFramework37
v ExampleofanEvaluationSituationandApproach38
TableofContents
vWhatWeMeanandWhyItMatters 6
vEvaluation&EvaluativeThinking 7
vSystemic&SystematicInquiry 9
vKnowledge&KnowledgeManagement10
vSystems 11vComplexSystems13vSystemsChange 21vSystemConditions22v ImplicitBias23vWhyItMatters25
01 02 03INTRODUCTION
Pages 34–41
Pages 26–33Pages 5–25
Pages 1–4 WhatWeMeanandWhyItMatters
AComplex-SystemsOrientationtoEvaluation
AComplex-SystemsOrientationtoEvaluationthatSupportsaCultureofHealth
RESOURCES
Page 42-43
v ImplicationsforEvaluationandEvaluators43
vReferences 45
vAcknowledgements46
Pages 44–46
CalltoAction
04
RW.17.rf.SystemsWhitePaper.2-14
Introduction
v ThePaper’sOrientationv ThePaper’sPurposev KeyPointsofthePaper
|1
ThePaper’sOrientation
Sinceevaluation’sinceptionasafieldinthe1960s,thefieldhasbeendominatedintheU.S.byan
implicitbiastowardhierarchicalstructuresthattendtobestable,predictable,andcontrollable.Such
controlleddynamicsarethebasisofhierarchiesandfactorymodelsoforganization.Inaboutthe
1980s,systemstheoristsrangingfrommathematiciansandphysiciststobiologists,sociologists,
psychologists,andecologistsbegantounderstandcomplexsystemsashavingnetworkedstructures
withemergentdynamicsthatinternallychange,formingflexible,complexnon-linearpatterns
(sometimescalled“self-organizing”dynamics).Thecapacityofcomputerswasbecomingsufficientto
modeltheseemergentpatternsinareasonableamountoftime.
Withtheexpandedcapacityofcomputersandsocialnetworks,evaluatorscanmorepowerfully
contributetounderstandinghowtoinfluencesocialconditionsforthepublicgoodbygroundingtheir
evaluationpracticeinthefundamentalsofcomplexsystems.Influencingsocialconditionsthrough
influencingcomplexsocialsystemsisattheheartofcreatingaCultureofHealth.
Introd
uctio
n
|4
ThePaper’sPurposeThisconceptpaperprovidesawaytothinkaboutcomplexsystems.Thepaper
countersevaluation’sbiastowardassumingthatanevaluationisoccurring
withinhierarchicalstructureswiththeircontrolledandstablepatternsof
movement(“dynamics”).Thisdominantapproachpaysinsufficientattention
totheemergentdynamicsofnetworkedstructuresthatcharacterizemany
systemsthatalsocontributetoacultureofhealth.
Theevaluationfield’stendencytofocuslargelyonhierarchicalstructuresis
evidenced,forexample,inprogramlogicmodelsthatfocusondesired
measurableoutcomesandtheprogram’sefficiencywithintheconfinesofa
hierarchicallystructuredsystem.Suchevaluationstypicallypaylimited
attentiontootherresultsoftheinterventionthatarenotwithinthecontrolof
theprogram.
Acomplex-systemsorientationshiftsevaluators’attentiontomultiplepatterns
overtimebasedontheintertwinednatureofhierarchicalandnetworked
structureswithincomplexsystems.Webeginbylookingatcomplexsystems
andthenreturntoacultureofhealthandsuggestsomeimplicationsfor
evaluatorsandevaluationusers.
Thespecificsofhowtoapplysuchanorientationtoevaluationareinneedof
conversationanddevelopment.Thispaperisintendedtostimulate
conversationsratherthanprovidefinalanswers.
Purpose:v Toencourageevaluativeinquiryofcomplexsystems
v Togroundevaluativeinquiryofcomplexsystemsin:~ understandinghowhierarchicalandnetworkedstructurescreatecomplexsystems~ recognizingevaluation’simplicitbiastowardhierarchicalsystems
Introd
uctio
n
|3
KeyPointsofthePaper
Acultureofhealthisacomplexsystem
Evaluationtosupportthecreationofacultureofhealthneedstobecongruentwiththe
fundamentalnatureofcomplexsystems
Dominantevaluationapproacheshaveanimplicitbiascountertoacomplexsystemsorientation
TheRobertWoodsJohnson
Foundation’sCultureofHealth
involvesfourresearch-basedaction
areas.Theyinteracttocreate
improvedpopulationhealth,well-
being,andequity.
Thefouractionareasare:
• makinghealthasharedvalue
• fosteringcross-sector
collaborationtoimprovewell-
being;
• creatinghealthier,moreequitable
communities;and
• strengtheningintegrationof
healthservices,andsystems.
Asevaluators,wecanassist
evaluationusersinseeing,
understanding,andinfluencingthe
developmentofacultureofhealth
byusingorientationsand
consequentmethodologiesthatare
congruentwiththenatureofthe
entangledstructuresanddynamics
ofcomplexsystems.
Thispaperilluminateswhyand
howtoleveragetwofundamentally
differentsystemstructures—
hierarchicalandnetworked—when
usingacomplex-systems
orientationinevaluation.
Today’sdominantevaluation
methodologicalorientationhas
animplicitbiastowardsystems
withhierarchicalstructures.This
dominantorientationfailsto
recognizetheimportantand
fundamentaldifferencesbetween
thehierarchicalandnetworked
structuresthatintertwinetoform
complexsystems.
Introd
uctio
n
|2
WhatWeMeanby:v Evaluation&EvaluativeThinking
v Systematic&SystematicInquiryv Knowledge&Knowledge
Managementv Systems
v ComplexSystems
v SystemsChangev SystemConditionsv ImplicitBias
01andWhyitMatters|5
WhatWeMeanandWhyItMattersWhatWeMeanby:v Evaluation&EvaluativeThinkingv Systemic&SystematicInquiryv Knowledge&Knowledge
Managementv Systemsv ComplexSystemsv SystemsChangev SystemConditionsv ImplicitBias
Oursocietyisdeeplyrootedinthinkingintermsof“stable
systems,”bywhichmostpeoplemeanhierarchicalstructures
withcontrolleddynamics.Tothinkaboutcomplexsystems,we
needtoaddnewtermsandconceptsandredefinesome
existingterms.
Thissectionexplainshowtothinkdifferentlyaboutconcepts
thatseemfamiliarbuthaveadifferentmeaninginthecontext
ofcomplexsystems.Itisespeciallyimportanttograspthe
differencesbetweenstructuresthattendtohavecontrolled,
stablepatternsofdynamics(hierarchicalstructures)andthose
thattendtohaveflexible,emergentdynamics(networked
structures). ??…[E]veryinstanceoflanguageusemakesitsownsmallcontributiontoreproducingand/ortransformingsocietyandculture,includingpowerrelations.(Fairclough&Wodak,1998).
01.W
hatW
eMeanandWhyItM
atters
|6
01.W
hatW
eMeanandWhyItM
atters
??Evaluation&EvaluativeThinking
Evaluation issystematic
inquiry to determinemerit
(quality),worth (value),and
significance (importance)
aboutsomeentityorpurpose.
AccordingtotheEncyclopediaofEvaluation:
Evaluation isanappliedinquiryprocessforcollectingandsynthesizing
evidencethatculminatesinconclusionsaboutthestateofaffairs,value,
merit,worth,significance,orqualityofaprogram,product,person,policy,
proposal,orplan.
Conclusionsmadeinevaluationsencompassbothanempiricalaspect(that
somethingisthecase)andanormativeaspect(judgementaboutthevalue
ofsomething).Itisthevaluefeaturethatdistinguishesevaluationfrom
othertypesofinquiry,suchasbasicscienceresearch,clinicalepidemiology,
investigativejournalism,orpublicpolling.” (Mathison,2005)
Ofparticularimportanceisthelaststatementthatemphasizesthat“itisthe
valuefeaturethatdistinguishesevaluationfromothertypesofinquiry.”
Thefield ofevaluationas practicedintheU.S.andmostothercountriesis
groundedinanassumptionthathumanshaveequalrights.Alsoevaluationis
donewithinrealworldsettings,notlaboratorysettingsdisconnectedfrom
thecontextinwhichtheentityistofunction.
Thusthedefinitionofevaluationdoesnotendwithsystematicinquiry,but
usesthisinquirytodeterminemerit,worth,and/orsignificance.
|7
01.W
hatW
eMeanandWhyItM
atters
EvaluativeThinkingExample
Evaluativethinkinginvolves
balancingmerit,worth,and
significance.
Forexample,assumeyouarelookingforanewcar.
Youfindanelectriccarthatisofveryhighquality(merit).Itisofhighvalue
(worth)toyoubecauseitalignswellwithyourcommitmentto
environmentalresponsibility.However,ifyourdaughtersuddenlybecomes
seriouslyillandyouneedtospendyourmoneyonhealthcareforher,the
cardropsinitsimportance(significance)becauseyourdaughterismore
importanttoyouthanacar.
Traditionally,evaluatorshavemadeevaluativedeterminationsbasedon
theirsystematicinquiryandpredeterminedcriteria.Asthecomplexityof
societyhasbecomemoreevident,evaluatorshavebeguntoengageafull
rangeofstakeholderswithmultipleperspectivesindeterminingmerit,
worth,andsignificance.Additionally,theyarelearningtohelppeople
balancemerit,worth,andsignificanceincomplexsystems.
??
|8
01.W
hatW
eMeanandWhyItM
atters
Systemic&SystematicInquiry
• Engageinsystematicand
systematic inquiry.
• Considercontent,context,conditions, andconnectivity.
• Useinterdisciplinary knowledge.
Therearemanydifferentmethodologiesforengaginginsystematic
inquiryinwaysthatarebothethicalandofhighquality.Wearenot
addressingtherangeofmethodologiesinthispaper;ratherthe
focusisondoinganinquiryfromasystemicperspective.
Threekeypointsforaninquiryare:
• Theevaluatorengagesinsystemic aswellassystematicinquiry.
• Inthinkingsystemicallyandsystematically,theevaluatorlooks
broadlyanddeeplyintothecontent,context,conditions,and
connectivityofthesystemsinvolved.
• Thetheoryofchangeguidingasystems-orientedevaluation
typicallyneedstorelyonastronginterdisciplinarybodyof
knowledge.
??
|9
01.W
hatW
eMeanandWhyItM
atters
Knowledge&KnowledgeManagement
Knowledgemanagement (KM)isthe
processofcreating,sharing,using
andmanagingtheknowledgeand
informationofanorganization.(Girard&Girard,2015)
Complexsystemsinvolvemulti-disciplinaryknowledge.Accessing
andsynthesizingabroadrangeofknowledgetoassistevaluation
usersindevelopinganappropriatetheoryofchangebecomesan
increasinglyimportantroleforevaluatorswhenworkingwith
complexsystems.Evaluatorsandevaluationuserstypicallyare
familiarwithamorelimitedrangeofdisciplinaryknowledgethan
whatisrelevantwhenworkingincomplexsystems.
Thispaperdoesnotaddressknowledgemanagementbutitisa
criticallyimportantaspectofengaginginsystems-oriented
evaluation.Fortunately,moretechnologically-supportedmeansfor
knowledgesynthesisandmanagementareemergingbutmuchmore
developmentisneededforefficientmanagementoftheknowledge
relevanttoseeing,understanding,andinfluencingcomplexsystems.
??
|10
01.W
hatW
eMeanandWhyItM
atters
SystemsAsystemisacollectionofentitiesthatare“seen”bysomeoneasinteractingtogethertodosomething.(OpenUniversity,2011)
Severalaspectsofthisdefinitionareimportant:
• Itisacollectionofentities;multipleelementsareinvolved.
• Differentpeoplemaybeseeingadifferentcollectionofentitiesinteractingtogether,thus,seeingadifferentsystem.
• Thequotationmarksaround“seen”remindusthatthesesystemsmaybeseeninourmind’seyeratherthannecessarilybeingseenphysically.Systemsaremostoftenconceptualmodelsratherthanphysicalentities.
• Theentitiesareinteracting.Therearebothconnectionsanddynamicsinthoseinteractions.Noticemovement andenergy.
• Theinteractingentitiesaredoingsomething.Theycreatearesultthatisdifferentthantheresultsofthesumoftheparts.
• Inevaluation,desiredoutcomesaretypicallythoughtofastheresultthatasystemproduces.However,thisistoonarrowofadefinitionofwhatasystemdoes.Asystemislikelytodomorethanproducetheoutcomessomeoneishopingtosee.Forexample,theresultcanbechangesinpatternswithinthesystems,notonlyspecificproducts.
??
System: Asystemisa
collectionofentitiesthatare
“seen”bysomeoneas
interacting togethertodo
something. (OpenUniversity,UK)
Systemsthinking:Thinkingin
termsofsystemsratherthan
theparts;thinkingabout
movement,dynamics,and
patterns acrosstimeand
locations;thinkinginsystems
|11
01.W
hatW
eMeanandWhyItM
atters
SystemsThinking
System: Asystemisacollection
ofentitiesthatare“seen”by
someoneasinteracting
togethertodosomething.(OpenUniversity,UK)
Systemsthinking:Thinkingin
termsofsystemsratherthan
theparts;thinkingabout
structures,dynamics,and
patterns acrosstimeand
locations;thinkinginsystems
Regardinghumanandotherlivingsystems,systemsthinkinginvolvesthinkingaboutstructures,dynamics,andpatternsacrosstimeandlocations.Sometimesitisreferredtoas“thinkinginsystems”.
Lookatthepicturesonthispage.Whatsystemsdoyou“see”?(Thepicturesmayshowonlyapartofasystem.)Forexample,thepictureoffoodinamarketmayleadyoutothinkofasystemof:
• transportationfromfarmtomarket
• theflowofmoneyrelatedtoagriculture
• theinteractionofpeopleatalocalfarmers’market
Whatsystemsdoyouseeinthispictureandtheotherpicturesbelow?
??
|12
01.W
hatW
eMeanandWhyItM
atters
ComplexSystems
Complexsystems:Complexsystems
aremassivelyentangledsystems that
includebothhierarchicaland
networkedstructures.
Complexsocialsystemsarecomposedofmassivelyentangledstructures.Forexample,theyinvolveinterconnected families; hierarchical,bureaucraticorganizations; andnetworksofsmallformalandinformalgroups. Notonlyaretheyentangled,theyarealsoinmotionwithcomplexpatterns.
Fromatheoreticalperspective,complexsystemscanbethoughtofasanentangledmixofhierarchicalandnetworkedstructures(seefollowingpages).Hierarchicalandnetworkedstructuresareassociatedwithdifferencesintheirdynamics,i.e.,theirpatternsofmovementovertime;theirpredictability;andtheirknowability.
Complexsystemshavebeenstudiedextensivelyindifferentdisciplineswithanaccumulationofknowledgeaboutthedifferencesinhowtheyfunction.TheReferencelistincludesafewsourcesthatwehavefoundhelpfulwhenapplyingsystemsconceptstoevaluation.
Understandingacomplexsystembearssimilaritiestocomprehendingthisopticalillusion,whichcontainsimagesofbothayoungwomanandanoldwoman.Bothimagesarewithinthepicturebutifoneisoverlydominantinourview,wecan’tseetheother.
Wewanttogettothepointwherewecanmoveourfocusbackandforthbetweenhierarchicalandnetworkedstructurestoseethecomplexsystemasawhole.
??
|13
C
ThinkinginComplexSystems
Lookagainatthepicturesyoulookedatonanearlierpage.Thinkabouttheentanglementsofmultiplesystemsthat
youcanseeinthepictures.
??
01.W
hatW
eMeanandWhyItM
atters
|14
HierarchicalStructuresAmajoradvancewiththeIndustrialRevolutionwasthedevelopmentoffactoriesandassemblylines.Thesewereoftenaccompaniedbyhierarchicalorganizationalstructurestomanagepeople.Theunderlyingassumptionisthatplanningisusefulbecausethereisapredictablerelationshipbetweenactivitiesandresults.Suchstructuresaredesignedtogiveconsiderablecontroltoanorganizationalleaderormanager.Thesestructuresarefurthercontrolledbyrulesandpolicies;theytendtohavepartsthatcanbereplaced.
Whenaskedtoidentifyasystem, wehavefoundthatpeopleusuallynamesometypeofhierarchicalstructure(e.g.,theeducation“system”).Hierarchicalstructuresarepervasiveingovernment,business,andthenonprofitorganizationworlds.Theyhavemanysubsystemswithinthemthataresimilarlystructured(e.g.,themanagementsystem, theaccountingsystem).
Whenconditionsarestableandpredictableandthefunctionofthestructureistoproducewell-definedproductsoroutcomes,thistypeofstructurecanworkwell.Forexample,wewantcontrolandstabilityin anorganization’s payrolldepartmenttoassuretimelyandaccuratepaychecks.
Theassumptionsofstabilityandpredictabilityserveasthebasisforthinkingthatoneprogramcanbereplacedbyanotherwithinasocialsystemandthat,ifeachpartisworkingwell,theoverallsystemwillalsobeworkingwell.
Hierarchicalstructurestendtocreate controlleddynamicswithstable,predictablepatterns.Theyhaveconstraintswithinoraroundthemthatcreatetheirpredictability.
??
Manytoolsandmethodsexistforunderstandingandinfluencinghierarchicalstructures.Thesetoolsandmethods,basedonlinearmodelsofcauseandeffect,includestrategicplanning,settingspecificoutcomes,andusingresearchdesignssuchasrandomizedcontrolledtrials(RCT).
01.W
hatW
eMeanandWhyItM
atters
|15
01.W
hatW
eMeanandWhyItM
atters
HierarchicalStructureswithinComplexSystems
|16
v FundamentalStructure:hierarchical,mechanistic
v Dynamic:controlled,stable,predictable,“regular,”causeandeffect
v Patterns:linear,repetitive,interchangeableparts
v Examples:bureaucracies,assemblylines
01.W
hatW
eMeanandWhyItM
atters
Thecomplexityscienceshavetheirbeginningscenturiesago,buttheideasstartedtoreceiveseriousattentioninthescientificcommunityinaboutthe1980s(Capra,2014).Theunderstandingandapplicationsoftheconceptsofnetworkedstructureswithemergent(or“self-organizing”)dynamicsarestillintheirearlystagesofdevelopment.Theconceptoriginatedinthephysical,biological,andecologicalsciences;ithasmathematicalunderpinnings.
Inanetworkedstructure,weseetheemergenceofnew,unexpectedprocessesandpatternsofmovement.Thesehigher-levelemergentdynamicsunexpectedlyresultfromtheactionsofamultiplicityofsmalloccurrenceswithinthesystem.Thesmalloccurrenceswerenotplannedtocreatetheneworder.Theemergenceofthenewisnotcontrolledbyasingleentity,butresultsfromsemi-independentinteractionsofmanyagents.Theyareadaptingtoeachotherandtheenvironmentasawhole.Theseinteractionscanformpatternsovertimeandlocations.Thosepatternsmaynotbevisibletotheagentsthemselvesandthetimingofthepatternsisoftenunpredictable.
Networkedstructuresmaybedisturbedbyexternalinfluencesbuttheytendtobewithoutexternalcontrol.NetworkedstructuressuchasthehumannetworksontheInternetdisplayemergentdynamics:theyareflexibleandadaptive;noleaderisincontrol.
Networkstructuresfunctionfarfromequilibrium.Theycancreateemergentdynamics withnew,unexpected,self-organizingpatterns,properties,and/orprocesses.
??NetworkedStructures
|17
01.W
hatW
eMeanandWhyItM
atters
Networkedstructures also:
• aresensitivetoinitialconditions.Incomplexsystems,verysmalldifferencesininitialconditionscanhaveadisproportionatelylargeimpactonfutureevents.
• areinfluencedbyiterativefeedback.Theagentsare“learning”fromoneanotherandthecontextastheygetsignalsfromotheragentsandmodifytheirbehavior.
• co-evolve.Co-evolutionreferstotheinterdependentevolutionoftwoormorestructureswithinalargerecologicalsystem.Cooperation,competition,andinterdependenceinrelationtothesamelimitedresourcescreatefeedbackamongtheentitiesthroughthestructuresinacomplexsystem.
• haveactiveboundaries.Asystem’sboundariesarecreatedbythestructureitself.Dynamicstructuressuchasanetwork’shaveactiveboundariesastheyinteractwiththeirenvironments.
• mayoperatefarfromequilibrium.Theycanhaveawiderangeofemergentdynamicswithpatternsofmovementthatmaybeverycomplex.
• canshiftintonewstructures.Theshiftintoaneworganizationalstructurecanarisefromfluctuationsamplifiedbycertaintypesoffeedbackloops.
Networkedstructurescancreateemergentdynamicswithnew,unexpected,self-organizingpatterns,properties,and/orprocesses.
??CharacteristicsofNetworkedStructures
|18
01.W
hatW
eMeanandWhyItM
atters
v Fundamentalstructure:networksofnodesandconnections
v Dynamic:flexible,creative,emergent
v Pattern:co-evolving,semi-independentagentsinfluencingnearbyagents
v Examples:networks,socialmovements,movementsofmovements,swarmingofbirds,schoolsoffish
NetworkedStructureswithinComplexSystems
|19
01.W
hatW
eMeanandWhyItM
atters
Seldomwouldanidentifiedsystemhavethepurefeaturesofahierarchicalornetworkedstructureasdescribedabove.Rathersocialsystemsareamixofstructuresalthoughtheymaytendinonedirectionortheotherintheiroverallcharacter.Hierarchicalandnetworkedstructuresarelensesthroughwhichweviewcomplexsystems.Eachlensrevealsdifferentaspectsofthecomplexityandgivesusinsightintoitsfunctioning.
Complexlivingsystemsgenerallyhavesomeaspectsthatcanbemodeledwellbythinkingofthemashierarchicaland/ornetworkedstructures.
Organizationalpartnershipsandcommunitiesaregoodillustrationsofcomplexsystemswherehierarchicalandnetworkedstructuresbothcanbeidentifiedfairlyreadily.
ComplexSystems:EntangledHierarchicalandNetworkedStructures
|20
01.W
hatW
eMeanandWhyItM
atters
SystemsChangeWe oftenhearpeopletalkaboutwantingtobringaboutchangeinsystemsthroughtheirinterventions.Whenengaginginevaluationrelatedtochangingsystems,weneedtobeveryawarethatthesystemsnaturallyarealreadyinmotionandchanging.Weneedtobeawarethattheinterventionisnotenteringastaticsystem.Interventionsarelikelytochangethepattern ofchangeratherthansimplycreatechange.
Humansystemsarecontinuallychanginginandofthemselvesandthroughtheirconnectionswithothersystems.Theyarelivingsystemsthatarecontinuallychangingwhetherornotthereisanintervention.Thisisespeciallytrueofsystemswithnetworkedstructures.
Interventionssuchasprograms,policies,people,andinitiativesarechangingthepatternsofchange thatarealreadypresentinthesystems.
Itisalsoimportanttorecognizethataninterventionmaybeaffectingmoresystemsthanthosefocusedonbytheintervenor.Thecomplexityoftheinterconnectedsystems,theunpredictabilityofpatternsofchangeinthesesystems,andthesustainabilityofchangesallneedtobetakenintoaccountwhenevaluatinganinterventionincomplexlivingsystems.
Complexlivingsystemsarecontinuallychangingbasedontheirinternaldynamicsaswellasoutsideinfluences.
The influenceofaninterventiondependsontheexistingstructuresanddynamicsofthesystemsbeingaffected.
Programs,policies,peopleandotherinterventionsinfluencethepatternsofchangethatarealreadypresentinthesystems.
??
|21
01.W
hatW
eMeanandWhyItM
atters
SystemConditionsSystemstheoristshaveidentifiedcommonfeaturesorconditionsofsystems.DrawingontheworkofWilliamsandHummelbrunner(2009)andEoyang(1997),weuseInterconnections,Boundaries,andPerspectivesasabasicwaytoframetheconditionsthatdeterminethenatureofhumansystemsandtheirspecialrelevancetoevaluation.Asweconductacomplex-systemsevaluation,weneedtopayspecialattentionto:
• Understandinginterconnections(interrelationships)
• Maintaininganawarenessofboundaries
• Recognizingmultipleperspectives
Interconnections,boundaries,andperspectivesarenotisolatedfromoneanother.Asyouchangeone,youaffectanother.Thinkofthethreeconditionsastheanglesofatriangle.Asyouchangeoneangleyouaffecttheothertwo.
Theseconditionsarerelevantatanyphaseofevaluation.Theycanalsobeusedtounderstandcomplexsystemswiththeirentangledcombinationsofhierarchicalandnetworkedstructures.
Understandingtheinterplayamongtheseconditionsispartofthechallengeofthinkinginsystems.Weneedtoapproachitwiththe“forestandthetrees”mindset.Wehavetobeabletoshiftinandoutoffocusbetweenseeingtheforestandseeingtheindividualtrees.
Thesesystemconditionscanbeusedtohelpyouidentifythestructuresandtheirdynamicsinthesystemsthatyouaregoingtobeevaluatingandchooseevaluationdesignsthatarecongruentwiththesestructures.
Humansystemshavethreegeneralcommonsystemconditions:
Interconnections
Boundaries
Perspectives
I
B P
??
|22
01.W
hatW
eMeanandWhyItM
atters
ImplicitBiasAnimplicitbiasisdifferentfromasuppressedthought thatsomeonemightconceal.Implicitbiasesare“biasesthatareactivatedinvoluntarilyandbeyondourawarenessorintentionalcontrol”(Staats,Capatosto,&Contractor,2015,p.5).Implicitbiasesoftenshowupinourlanguage.
“Unwantedimplicitbiasescanbemitigated….Bytakingthetimetounderstandyourpersonalbiases,youcanbegintomitigatetheireffects....Implicitbiaseshavehugeimplications;thusitisimportanttoidentifyyourownbiasesandthenactivelyengageindebiasingtechniquestoaddressthem.” (Staats,Capatosto,&Contractor,2015,p.5).
Thereisgrowingattentiontostructuralracismastheresultofimplicitbiases.Structuralracismreferstothenormalizationandlegitimizationofinstitutionalpoliciesandpracticesthatroutinelyadvantagewhiteswhileproducingcumulativeandchronicadverseoutcomesforpeopleofcolor.Structuralracismbecomesasystemofitsownduetotheseinterconnectedorganizedsystems.
Theterm“implicitbias”canalsobeappliedtowhatishappeninginevaluationeventhoughtheimpactisverydifferentthanthatresultingfromstructuralracism.
Whenitcomestoevaluation,acriticallyimportantimplicitbiasistowardamodelofsystemswithhierarchicalstructuresandcontrolleddynamics.Thispervasivebiashasmanyimplicationsforevaluation.
“Implicitbiasisconcernedwithunconsciouscognitionthatinfluencesunder-standing,actions,anddecisions”. Thisisdifferentfromindividualsconsciouslysuppressingornotsharingtheirbeliefsbecauseothersmayviewtheirbeliefsassociallyundesirable.
??
|23
01.W
hatW
eMeanandWhyItM
atters
Evaluation’sImplicitBias
Evaluation’simplicitbiastowardsahierarchicalormechanisticorientationisalignedwiththewayofthinkingthathasdominatedoursocietysincetheIndustrialRevolutionwhensystemswerethoughtofasstable.Considerthefactorywhereanassemblylineofworkersfitspartstogethertocreateafinalproduct.Themechanisticmodelisalsoreinforcedbybusiness,whichseeksahigh-qualityoutcomethroughefficiencyandeffectiveness.Thehigh-qualityproduct/outcomesbecomestheaimofthework.
There’sawholeschoolofdesignthatcomesoutofthisbusinessmodel.Inthiskindofdesign,peoplegooutintothecommunitytofindoutthepeople’sneeds.Theyaregoodatinvestigatingpeople’slivesinfriendly,non-obtrusiveways.Theygettoknowaculturewiththepurposeofcomingupwithideasaboutproblemsthatcouldbesolvedwithaproductthattheycandevelop.Theydonotidentifyissuesthatcanbesolvedbypeoplesimplyworkingorlivingtogetherdifferentlywithoutmoreproducts.Instead,it’saboutfindingsomethingthatcouldbethebasisofaproductthebusinesscanselltothepeople.Thatmodelofdesignismovingintoevaluationwithoutrecognizingthatthedesignhasanimplicitbiastowardtheproductionofsomething.
Eventhoughevaluationhasapproachessuchasdevelopmentalevaluationandparticipatoryevaluation,evaluators’implicitbiastowardlinearityleadstoviewingnetworkedstructureswithincomplexsystemsasanadd-ontothehierarchicalstructureratherthanasasystemstructureinitsownright.
Evaluationtendstohaveanimplicitbias,unrecognizedbyevaluatorsandevaluationusers,towardassumingsystemschangeinalinearfashion.
Takingacomplexsystemsorientationwillbeoflittleconsequenceunlessweconfrontthepowerandsignificanceoftheimplicitbiaswehavetowardalinearsystemsmodel.
??
|24
01.W
hatW
eMeanandWhyItM
atters
WhyitMattersInsummary,becauseofthetendencyinthefieldofevaluationtofocus
onhierarchicalstructureswithcontrolleddynamicsand/ortobe
unawareofcomplexitysciences,thissectionhasprovidedasetof
vocabularyandconceptsthatareimportantforevaluatorsto
understand.Withoutsuchunderstanding,evaluatorstendtouse
methodologiesthataregroundedinmechanistic(linearmathematical)
modelsthatarecongruentwithhierarchicalstructuresratherthan
developingmodelswherehierarchicalstructuresareentwinedwith
dynamicnetworkstructures.
??
|25
AComplex-SystemsOrientationtoEvaluation
v AComplex-SystemsOrientationtoEvaluation
v EvaluationGroundedinHierarchically-StructuredSystems
v EvaluationGroundedinNetwork-StructuredSystems
v ContrastofEvaluativeFocusinHierarchicalandNetworkedStructures 02
|26
AComplex-SystemsOrientationtoEvaluationAsystemisacollectionofentitiesthatare‘seen’bysomeoneasinteractingtogethertodosomething.Complexsystemsaremassivelyentangledsystems thatincludebothhierarchicalandnetworkedstructures.
Thissectionofthepapershiftsthefocustotakingacomplex-
systemsorientationtoevaluation.Thatis,asevaluators,weaccept
thatweareworkingwithawebofentangledstructureswithmany
gradationsofdynamicsthatrangefromcontrolledandstableto
emergentandflexible.Withinthisdynamicweb,weworkwith
evaluationuserstoidentifythesystemswithinthiswebthatwe
aregoingtoinvestigatethroughtheevaluation.
Recallourearlierdefinitionofasystem:“Asystemisacollectionof
entitiesthatare‘seen’bysomeoneasinteractingtogethertodo
something.”Thus,ourinitialstepinconductingacomplex-systems
orientedevaluationistointeractivelydevelopapicture(literallyor
figuratively)ofthemultiplicityofhierarchicalandnetworked
structuresasseenbyanarrayofstakeholdersrelevanttothe
work.Fromthere,wehighlightandputourfocusonthose
deemedtobethemostimportantstartingpointforthe
evaluation.Wewillnotbediscussingthedetailsofhowtoengage
insuchdeterminationsinthispaper.
|27
02.ACom
plex-SystemsO
rientationtoEvaluation
02.ACom
plex-SystemsO
rientationtoEvaluation
Complex-SystemsOrientedEvaluationDesign
Theevaluatorlooksforthepresenceofthetwobasicsystemstructures—hierarchies(controlled
dynamics)andnetworks(emergentdynamics)—inthesystemsemphasizedintheevaluation.The
evaluatorconsiderstheimplicationsofthestructures andtheirdynamicsfortheevaluation.
(Althoughtherearemanyotheraspectsofcomplexsystemsthatwecouldaddress,thisisourfocus
inthispaper.)
Wealigntheevaluationdesignwiththefundamentalstructures.Asnotedintheabovediscussion,
anygivenevaluationislikelytohaveamixofmethodsandorientation.However,atthispointwe
wanttoclarifythesystemstructuresinwhichtheevaluationisfundamentally“grounded.”
• Istheevaluationgroundedprimarilyinaffectingthehierarchicalstructureswithinwhichthe
interventionislocated?
AND/OR
• Istheevaluationgroundedininfluencingthedynamicnetworksofnodesandconnections?
|28
EvaluationGroundedinHierarchicalStructures
Whenthefundamentalstructureisahierarchicalone,theevaluationemphasisismostlikelygoingtobeonfeaturesofhierarchicalstructuressuchasprograms,policies,andpracticeswithintherelevantsystems.Althoughadaptationofaninterventionmaybeamajorareaofconcernandthesystematicinquiryprocessitselfmayconsumethemajoramountoftime,theevaluative aspectislikelytoemphasize:
v Measuringoutcomesv Linkingoutcomesandactivitiesv Applyingpredeterminedevaluationcriteriav Buildingevaluationcapacityaroundtechnicalaspectsofdeterminingoutcomesandevaluation
criteria
|29
02.ACom
plex-SystemsO
rientationtoEvaluation
EvaluationGroundedinNetworkedStructures
Whenthefundamentalstructureisanetwork,theevaluationfocusesonthefundamentalfeaturesofnetworks—nodesandconnections.Thesenodesandconnectionsmaywellincludehierarchicalfeaturesbuttheemphasisisonthenetworkedaspects.Thustheevaluative aspectislikelytoemphasize:
v Mappingpatternsofmovementandchangerelatedtovaluesv Facilitatingconversationsaboutmerit,worth,andsignificancev Buildingthecapacityofnetworkstosustaincontextuallyappropriateattentiontomerit,worth,and
significanceamongmultiplestakeholders
|30
02.ACom
plex-SystemsO
rientationtoEvaluation
Neighborhood
Faith-BasedOrganization
SocialMovement
GovernmentalAgency
NationalCommunityLeadershipNetwork
ExampleofEvaluationGroundedinNetworked-StructuresAssumethattheinterventionofinterestisanationalnetworktosupportacultureofhealth.Iftheevaluationistobegroundedinnetworkedstructures,theevaluationmayfocusonfollowingvariousmembersofthenationalnetworkastheyworkthroughnetworksintheircommunitylocation.Thediagrambelowillustratesthenetworksofonememberofanationalnetwork.Thenetworkontheleftisaportionofthelocalcommunity;onememberofthenetworkisbeingfollowedintohercommunity.Therestofthediagramshowsthatpersoninherlocalnetworks.
|31
02.ACom
plex-SystemsO
rientationtoEvaluation
Whentheevaluatorofthenationalnetworkconductsanevaluativefacilitationandinquiryinthisnetworkmember’scommunity,shemightfocusherevaluationonequity,leadership,andpowerdifferentialsinthevariousstructuresinthecommunityasillustratedbelow.
Neighborhood
Faith-BasedOrganization
SocialMovement
GovernmentalAgency
NationalCommunityLeadershipNetwork
Inquiryfocusesonunderstandingdifferencesinthe
powerofhierarchicaland
networkedstructures.
ExampleofEvaluationGroundedinNetworked-Structures
|32
02.ACom
plex-SystemsO
rientationtoEvaluation
v Mappatternsofmovementandchangerelatedtovalues
v Facilitateconversationsaboutmerit,worth,andsignificance
v Buildcapacityofnetworkstosustaincontextuallyappropriateattentiontomerit,worth,andsignificance
HierarchicalStructures NetworkedStructures
v Measureoutcomesv Linkoutcomesandactivitiesv Applyevaluationcriteriav Buildevaluationcapacityaboutaspectsofsystematicinquiry
ContrastofEvaluativeFocusinHierarchicalandNetworkedStructures
|33
02.ACom
plex-SystemsO
rientationtoEvaluation
AComplex-SystemsOrientationtoEvaluationthatSupportsa
CultureofHealthv Culturev ACultureofHealthv ACultureofHealthActionFrameworkv ExampleofanEvaluationSituationandApproach
§ ExampleEvaluationBackground§ ExampleEvaluationApproach§ ExampleEvaluationQuestions:Interconnections,Boundaries,
Perspectives 03|34
CultureCultureisdefinedinanumberofways.Herearethree:
• Cultureisasharedsetofvalues,customs,andbeliefscommontoaparticulargrouporsocietythatproducespatternsofbehaviorsandexperiences.
• Cultureasmultiplediscourses,occasionallycomingtogetherinlargesystemicconfiguration,butmoreoftencoexistingwithindynamicfieldsofinteractionandconflict.(Dirks,Eley,&Ortner,1994,p.4)
• Incontrastwiththeclassicview,whichpositscultureasaself-containedwholemadeupofcoherentpatterns,culturecanarguablybeconceivedasamoreporousarrayofintersectionswheredistinctprocessescrisscrossfromwithinandbeyonditsborders.(Rosaldo,1993,p.20)
Allsaid,cultureisacomplexsystemwith massivelyentangledentitiesandhierarchicalandnetworkedstructures.Itsboundariesarepermeableanditisdispersedthroughsociety.Someculturalaspectsareimposed byoutsidegroupsorindividuals whileothersarisefromwithinthegroup.
Cultureisoftenthoughtofintermsofracialorethnicgroups.Cultureemergesinmanygroups.Abusiness,neighborhood,community,sportsteam,family,ornetworkdevelopsitscultureovertime.Thesharedvalues,customs,andbeliefsmayormaynotbeexplicitandamemberofthegroupmaybeconsciousorunconsciousoftheculture.
Culture isacomplexsystemwithmassivelyentangledentitiesandhierarchicalandnetworkedstructures.
|35
03.ACom
plex-SystemsO
rientationtoEvaluationthatSup
portsa
CultureofH
ealth
03.ACom
plex-SystemsO
rientationtoEvaluationthatSup
portsa
CultureofH
ealth
ACultureofHealthTheaimistofocusonbuildingacultureofhealthratherthan
focusingonlyonhealthcareservices.
Toaccomplishthis,changesmustbemade
inbeliefsandassumptions,andthe
structuresofsocialsystems.
“Recognizethathealthandwell-beingcanbegreatlyinfluencedbycomplexsocialfactors:wherewelive,howwework,thesoundnessandsafetyofoursurroundings,andthestrengthandresilienceofourfamiliesandcommunities”*
“ComprehensiveCultureofHealth…enablesallinourdiversesocietytoleadhealthierlives,nowandforgenerationstocome”*
“Creatingasocietythatgivesallindividualsanequalopportunitytolivethehealthiestlivespossible,whatevertheirethnic,geographic,racial,socioeconomic,orphysicalcircumstanceshappentobe”*
RWJFisfocusedonbuildingacultureofhealththatemphasizesequityandanunderstandingthathealthisdeterminedbymanysocialconditions.BylabelingtheirworkasCulture ofHealth,RWJFrecognizestheimportanceofhealthbeingbasedonsharedvalues,customs,andbeliefsandhowtheseneedtochangewithinvariousgroupstohavepopulationhealth,well-being,andequity.RWJF’sapproachcontraststoonethatfocusesonlyonhealth-careservices.Thehealth-care-serviceapproachassumesthatindividualsarelargelyresponsible fortheirownhealth;itignorestheinequitiesandtheconfusingprocessesandmultipleorientationswithinhealthcareandothersocialdeterminantsofhealth.RWJF’sapproachfocusesonpeoplechangingtheirbeliefsandassumptionsandthestructuresoftheirsocialsystems,i.e.,changingtheculturetoaCultureofHealth.
*Copyright2015RobertWoodJohnsonFoundation/FromVisiontoAction:AFrameworkandMeasurestoMobilizeaCultureofHealth.
|36
ACultureofHealthActionFrameworkBasedonthesegeneralbeliefsandassumptions,theFoundationhasidentifiedfourareasforactiontobringaboutthecultureshift.Thefoundationhasalsostatedanencompassingoutcomethatwouldresultfromthisshift.
Notethattheframeworkisbroad.Itdoesnotgivespecificactionsforpeopletotakebutprovidestheareasonwhichtofocusandincorporatesthebeliefsthatarekeytothenewculture.Itisgivingpeopleatanyplaceinsocietyareasinwhichtofocus.
Buildinganewculturemaybelessaboutactionswedoandmoreaboutadialoguethatflowsthroughandamongus.
Imagecopyright2015RobertWoodJohnsonFoundation/FromVisiontoAction:AFrameworkandMeasurestoMobilizeaCultureofHealth. |37
03.ACom
plex-SystemsO
rientationtoEvaluationthatSup
portsa
CultureofH
ealth
ExampleofanEvaluationSituationandApproach Acoalitionthatrecentlyreceivedagranthasrequestedsuggestionsabout
conductingitsevaluationtodetermineifthemanysystemsinvolvedintheirworkareactuallyfunctioninginawaythatchangesthecultureregardinghealth.TheyareusingtheRWJFCultureofHealthActionFrameworkastheirguide.
Theexampleprovidessuggestionsrelatedtothegeneralevaluationapproachandevaluationquestions.
Here’sthegranteesituation:
Overa20-yearperiod,acoalitionhasguidedthelocatingofawiderangeofhealth-careservicesinonehealth-carecomplex.Thecoalition,locatedinalow-incomeregionofasouthwesternstate,includesvariousentitiessuchasnonprofitorganizations,health-careserviceproviders,andresidentsfromtheurbanneighborhoodsandruralareas.Severalyearsago,withcommunityinput,thecoalitiondevelopedastatementofsharedvalues.
Acommunitysurveyrevealedaneedforaccesstomorehealthyfood.So,thecoalitionisnowplanningtoaddagrocerystoreinthehealth-carecomplex.
Thecoalitionmembershaveanunderstandingofcomplexsystemsfromarecentroundofseminars.Theyareparticularlyinterestedinthenodesandconnectionsofthenetworksandwebsoftheircommunityandwhetherthey,asmembersofacoalition,areanimportantnodeinbuildinganeconomically-equitablecommunity.Theyarealsoawarethatcommunityorganizerswhoarenewmembersofthecoalitionthinkthatthecommunitysurveyprovidedaninadequateunderstandingofthecommunityasawhole.Photos(exceptfarmworkers)copyright2015RobertWood
JohnsonFoundation/FromVisiontoAction:AFrameworkandMeasurestoMobilizeaCultureofHealth;photosbyTyroneTurner.Farmworkers’photosretrievedfrompublicdomain. |38
03.ACom
plex-SystemsO
rientationtoEvaluationthatSup
portsa
CultureofH
ealth
ExampleEvaluationSituation
Coalitionmembersarepleasedwiththeprogresstheyhavemadeonoverallindicatorsofpopulationwell-beingandhealth,buttheyarenotsureifdeepchangesintheculture(changesinvalues,beliefs,andassumptions)areoccurringthatwillsupportasustainablecultureofhealthasdescribedbytheRWJFCultureofHealthactionframework.
Thecoalitionhasworkedcollaborativelyinthehealthcomplextobuildawidespreadunderstandingofthecentralityofhealthinthegovernmentalagencies,businesses,schools,anduniversities.Health-careserviceprovidersareintegratingtheirtechnologicalsystemstosharedataandmakeiteasierforresidentstoworkwithmultipleproviders.Theynowrecognizethattheseeffortsareallfocusedonhierarchicalstructures.
Theyareuneasyaboutwhethersufficientattentionisbeinggiventobuildinganeconomically-equitablecommunity,astructurethatismoregroundedinnetworks.
Intheirnextphaseofevaluationtheywanttofocusspecificallyonthefunctioningoftheircoalitionasanodewithinthemanyinterconnectednetworksandwebsoftheircommunitythatareespeciallyrelevanttobuildinganeconomically- equitablecommunity.
Photos(exceptfarmworkers)copyright2015RobertWoodJohnsonFoundation/FromVisiontoAction:AFrameworkandMeasurestoMobilizeaCultureofHealth;photosbyTyroneTurner.Farmworkers’photosretrievedfrompublicdomain.
|39
03.ACom
plex-SystemsO
rientationtoEvaluationthatSup
portsa
CultureofH
ealth
Workingwithasystems-orientedevaluatorfromanonprofitorganizationthatisamemberofthecoalition,thecoalitionmembersbeganbyidentifyingtheseguidelinesforconductingasystems-orientedandnetwork-focusedevaluation.
• Usebroadlydefinedboundaries fordeterminingwhattypesofnetworksandwebsexistinthecommunitythatmayberelevant.
• Attendtothe interconnections (especiallypowerrelationships) amongthefullrangeofstakeholders(individualsandcollectiveentities)inthesituation.
• Lookforpatternsofinterconnectednodes;lookformanytypesofresources;andlookathowthesevariousresourcesflowthroughthenetworkandareexchangedinthenodesofthenetworks.
• Encouragetransparencyofperspectives (values,beliefs,andassumptions)inallphasesoftheevaluation.
• Emphasizefacilitationthatdeterminesmerit,worth,andsignificancethroughouttheprocessasitrelatestohealthforall.
• Focusonaskingthe“right”questionsforpurposesofsharedreflectionandinquiryratherthannecessarilyansweringthequestionsinafinalsense.
• Emphasizebuildingthecapacityofcoalitionmemberstoengageinevaluative,reflectiveinquirythathelpsthemexpendtheleastefforttobuildtheself-organizingcapacityofthenetworkstosupportaneconomically-equitablecommunity.
• Drawonexpertisefrommultipledisciplinesrelevanttothesituation.
ExampleEvaluationApproach
|40
03.ACom
plex-SystemsO
rientationtoEvaluationthatSup
portsa
CultureofH
ealth
Herearesomeofthequestionstheyareconsidering.Theyasksomequestionstounderstandthecurrentandpastinteractionsabouttheeconomicallyequitablenatureoftheircommunityandhowtheestablishmentofagrocerystoreinthehealthcarecomplexmightrelatetothisdesiredcondition.Theyaskotherquestionstounderstandthebeliefs,assumptions,perspectives,andperceptionsaboutthemeaningandnatureofaneconomicallyequitablecommunity.
Interconnections
• Whatpatternsofpowerrelationshipsarepresentandhowaretheyshifting?
• Whatistheebbandflowofchangesbetweentheformalhealthorganizationsandtheresidentsofthecommunity?
• Whatshiftsareoccurringintheflowofinformation?
Boundaries
• Whatboundarieswithinandacrossorganizationsandwiththecommunityanditsneighborhoodgroupsareshifting?
• Whatdifferencesareevidentinthepermeabilityoftheboundarieswithinandacrossorganizationsandwiththecommunityanditsneighborhoods?
• Whatshiftsareoccurringinwhoisinvolvedandinwhatwaysinvarioustypesofdecision-making?
• Howarepatternsofactionandthoughtshiftingregardinghealthacrossgenerations?Whoisincluded?Whoisnot?
Perspectives
• Whatshiftsinvalues,beliefs,andassumptionsareoccurringrelatedtoequityanddemocracy?
• Whatshiftsinvalues,beliefs,andassumptionsareoccurringregardingtheimportanceofhealth?
• Whatshiftsinvalues,beliefs,andassumptionsareoccurringregardingwhatconstituteshealth?
ExampleEvaluationQuestions:Interconnections,Boundaries,andPerspectives
|41
03.ACom
plex-SystemsO
rientationtoEvaluationthatSup
portsa
CultureofH
ealth
CalltoAction
v ImplicationsforEvaluationandEvaluators
04|42
04.CalltoActio
n
ImplicationsforEvaluationandEvaluatorsThispapercallsonevaluatorsandevaluationuserstoattendtotwofundamentalstructuresofcomplex
socialsystems—hierarchiesandnetworks.Asastarter,considerthefollowing:
• Engageinevaluationwithattentiontobothhierarchicalstructureswithcontrolleddynamicsand
networkedstructureswithemergentdynamics.
• Recognizethatevaluationcapacitybuildingislikelytolookdifferentwhenworkingwithhierarchical
structuresversusnetworkedstructures.
• Payspecialattentiontoevaluativefacilitationaswellasinquiryprocesses.
• Recognizethatoutcomesinahierarchicalstructurearelikelytobedefinableproducts,knowledge,or
services,whereasoutcomesinanetworkedstructurearelikelytobechangesinpatternsofactionand
thinkingacrosstimeandlocations.
• Recognizethedifferenceinrolesofthetwotypesofstructuresinsustaining,restoring,andregenerating
society.
• Engageinconversationswithcolleaguesabouttheapplicationofthesesystemsconceptsandsharewith
uswhatyouarelearning.
|43
Resources
v Referencesv Acknowledgements
|44
ReferencesCapra,F.&Luisi,P.(2014).Thesystemsviewoflife:Aunifyingvision.Cambridge,MA:CambridgeUniversityPress.Dirks,N.B.,Eley,G.,&Ortner,S.B.(1994).Introduction.InN.B.Dirks,G.Eley,&S.B.Ortner(Eds.),Culture/power/history:Areaderin
contemporarysocialtheory(pp.3-45).Princeton,NJ:PrincetonUniversityPress.Eoyang,G.(1997).Conditionsforself-organizinginhumansystems.Unpublisheddoctoraldissertation,UnionInstituteandUniversity,
Cincinnati,OH.Fairclough,N.andWodak,R.,(1998)."CriticalDiscourseAnalysis"inDijk,T.A.(Ed.)(1998)DiscourseasSocialInteraction:Discoursestudies:A
multidisciplinaryIntroduction(Vol.2).LondonSAGE.Girard,JohnP.&Girard,JoAnnL.(2015)."Definingknowledgemanagement:Towardanappliedcompendium"(PDF). OnlineJournalofApplied
KnowledgeManagement.3 (1):1–20.Jackson,M.C.(2003).Systemsthinking:Creativeholismformanagers.WestSussex,England:JohnWileyandSonsLtd.KelloggFoundation,W.K.(2007).Designinginitiativeevaluation:Asystems-orientedframeworkforevaluatingsocialchangeefforts. Battle
Creek,MI:W.K.KelloggFoundation.Mathison,S.(2005).Encyclopediaofevaluation.ThousandOaks,CA:SAGEPublications.Meadows,D.H.(2008).Thinkinginsystems.WhiteRiverJunction,VT:ChelseaGreenPublishing.OpenUniversity.(2011).Systemsthinkingandpractice.Retrievedfromhttp://www.open.edu/openlearn/science-maths-
technology/computing-andict/systems-computer/systems-thinking-and-practice/content-section-0Parsons,B.(2012).Usingcomplexityscienceconceptswhendesigningsysteminterventionsandevaluations(3rded.).Retrievedfrom
http://insites.org/resources/using-complexity-science-concepts-when-designing- system-interventions-and-evaluations/Ramage,M.,&Shipp,K.(2009).Systemsthinkers.London,England:Springer.Rosaldo,R.(1993).Culture&truth:Theremakingofsocialanalysis.Boston,MA:BeaconPress.Scriven,M.(2013).Thefoundationandfutureofevaluation.InThefutureofevaluationinsociety:AtributetoMichaelScriven.Donaldson,S.
(Ed.).Charlotte,NC:InformationAgePublishing,Inc.Staats,C.,Capatosto,K.,&Contractor,D.(2015).Stateofthescience:ImplicitBiasReview2015. Columbus,Ohio:KilwanInstitutefortheStudy
ofRaceandEthnicity.Williams,B.&Hummelbrunner,R.(2009).Systemconceptsinaction:Apractitioners’toolkit. Stanford,CA:StanfordUniversityPress.Zimmerman,B.,Lindberg,C.,&Plsek,P.(2001).Edgeware:Insightsfromcomplexityscienceforhealthcareleaders. Irving,TX:VHA.
Websiteshttp://www.insites.org/
http://www.fsg.org/http://www.evaluationinnovation.org/
https://www.aspeninstitute.org/
Resources
|45
AcknowledgementsBeverlyParsons,PatriciaJessup,andMarahMoorewrotethispaperwithsupport
fromtheRobertWoodJohnsonFoundation.Theviewsexpressedheredonot
necessarilyreflecttheviewsoftheFoundation.
WewishtoexpressourthankstoLauraLevitonoftheRobertWoodJohnson
Foundationforherinsights,review,andfeedbackthroughoutthecompletionof
thegrant.
OurappreciationisalsoexpressedtoourcolleaguesintheAmericanEvaluation
Association’sSystemsinEvaluationTopicalInterestGroupforthemany
conversationsthathaveinformedthethinkingreflectedinthispaper.
WearealsodeeplygratefultoRosemaryReinhartforhereditingandKathy
Wyckoffforherformattinganddesignofthepaper.
Resources
|46October2016
Top Related