How complexity influences evaluationA presentation to the Australasian Evaluation Society Conference

Sydney, August 2011

Julie McGeary

Presentation outline

• Complex systems• Complex interventions• Role of program theory in complex evaluations• Audience expectations• Future Farming Strategy evaluation planning

• Non linear• Emergent• Dynamical• Adaptive• Uncertain• Co-evolutionary

Complex systems

Patton, M. (2010). Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. Guilford Press, NY, p. 8.

Victorian agriculture system

• Governance and implementation• Simultaneous causal strands• Alternative causal strands• Non-linearity and disproportionate outcomes• Emergent outcomes

Complex interventions

Rogers, P. (2008). Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation, 14(1), p. 32.

The Future Farming Strategy (FFS)

Interactions

Complex interventions are events in systems.

It is not the intervention itself but the context or setting into which it is introduced and with which it interacts which is the most significant aspect of the complexity.

Hawe, P., Shiell, A. and Riley, T. (2009). Theorising interventions as events in systems. American Journal of Community Psychology, 43, 267-276.

Role of program theory

Role of program theory

Inputs Outputs

Transformational potential

Future Farming Strategy logic model

Audience expectations

•Judgements of adequacy•‘Global’ net benefits•Causal inference – level of confidence•Quantification

“The tension between the systematic evaluation approaches proposed by evaluation experts and the impact measurements required by clients for accountability purposes is not going to be solved easily.”Molas-Gallart, J. and Davies, A. (2006). Toward theory-led evaluation: The experience of European science, technology, and innovation policies. American Journal of Evaluation, 27(1), p. 78.

Evaluation planning

Purpose

• Examine if FFS has made progress towards improving the productive, competitive and sustainable status of Victorian farm businesses

• Assess the benefits of Victorian Government investment in the FFS

• Provide opportunities for learning and improvement in whole-of-government strategy implementation and delivery to guide future policy and program development.

Evaluation planning

Implications of complexity

• Outcomes definition

• Alignment between Action and strategy level outcomes

• Forms of evidence

• Emergent evaluation

Everything should be made as simple as possible, but not simpler.

Albert Einstein, 1879 – 1955

Reality is complex. Nicoletta Stame, 2004

The art of dealing with the complicated and complex real world lies in knowing when to simplify and when, and how, to complicate.

Patricia Rogers, 2008

. . . and finally . . .