Influencing the use of evaluation findings

How Four State Government Organisations Are Working Towards Developing a Utilisation Culture

Introducing the Panel

•Anna Georgalis (Deborrah Jepsen) – Department of Business & Innovation

•Quynh-Tram Trinh – Department of Human Services

•Sue Doyle – Arts Victoria (Department Premier & Cabinet)

•Kirsty Fenton – Department of Primary Industries

Today’s Presentation will focus on:•Developing an organisational culture of evaluation through utilisation of evaluations

•Using evaluation evidence to support unpopular decision

•Factors and approaches that support the utilisation and influence of evaluation findings, such as:

• Methods that promote ownership of evaluations to increase the likelihood of implementation of findings and recommendations

• Establishment of a strategic evaluation agenda• Organisational maturity

Structure

Panel members – 30 mins

Questions & Discussion – 30 mins

Sharing approaches, challenges, ‘hot tips’ for developing a culture that supports the utilisation and influence of evaluation at all levels of an organisation

• Department of Business and Innovation

• Anna Georgalis

• Victoria’s lead economic, business and innovation agency

• Role is to continue to develop and promote policies and actions that increase productivity and participation which supports sustainable growth

Department of Business and Innovation (Victoria)

About the Victorian Department of Business and Innovation

Department of Business and Innovation (Victoria)

• Non-recurrent nature of program funding required recurring business case development

• No evidence to support why the initiative was important

• Increasing demand for accountability and transparency from:

– Ministers

– Victorian Auditor General

– Central Agencies

• New recent wave of strengthened focus from Central Agencies

The drivers for evaluation

Department of Business and Innovation (Victoria)

• Departmental Evaluation Framework and Policy that specifies:

– Roles and responsibilities

– Quality assurance

– Processes and timing

• An Initiative Design Framework for business case development that includes a focus on evaluation at the design phase

• The use of Investment Logic Maps (or equivalent) and development of evaluation or performance monitoring frameworks

• A central evaluation unit dedicated to supporting evaluation across the Department

Key approaches and tools in developing an organisational evaluation culture

Department of Business and Innovation (Victoria)How evaluation influences the development of strategies, policies and programs

Continuous Improvement Cycle

Department of Business and Innovation (Victoria)

• Increased understanding, capability and utilisation of evaluation across the Department

• Stronger business cases

• Relevant and appropriate new programs based on evaluations and stronger evidence

• Evaluation used to redesign programs or cease programs as necessary

• Culture of continuous improvement

• Increased accountability and transparency

Successes so far

Department of Business and Innovation (Victoria)

• Must have leadership direction from the top, authorising environment and critical mass of supporters

• Ensure engagement with stakeholders throughout

• Use authorising environment to your advantage

• Rigorous and ‘fit-for-purpose’ methodologies for evaluation

• Critical thinkers and analytical capability

• Sustained vigilance

• Bringing about change, especially using evaluation, is challenging

Lessons learnt

Evaluation & Influence in Government

Quynh-Tram Trinh, Manager, Research & Evaluation Unit

Department of Human Services, Victoria

12

Presentation outline

1.Centralisation of Evaluation Capacity

2.Authorisation for central R&E unit

3.Aspects of Influence

13

Centralisation of Evaluation Capacity

July 2010 – R&E Unit nominally centralised. It pulls together existing evaluation EFTs in:

• Housing & Community Building division (social housing & homelessness)

• Children, Youth & Family Services division (child protection, youth justice and family services)

• Small amount of additional resourcing in research

14

Authorisation for central R&E unitThe R&E Unit is authorised to:

• To provide strategic R&E leadership and improve the evidence base to support effective policy, program and service development in DHS; &

• To promote a collaborative and strategic approach to R&E in areas of priority for DHS.

R&E Unit sits in Policy & Client Outcomes Branch, IWS Division of DHS, Victoria. P&CO branch’s objective is:

To improve outcomes for people who many need the support of the human services system through high quality strategic policy,

research & evaluation

15

Aspects of Influence

• Day-to-day

• Strategic

16

Aspects of Influence – Day-to-day

• Active advice on high-quality evaluation approach e.g. objectives, questions, methodology, contract management, public dissemination

• Direct contract management on high-profile evaluations. Practise high-standard stakeholder management, contract management, ensure methodological rigour, good communications & public dissemination

17

Aspects of Influence – Day-to-day

• Facilitation of logic mapping sessions to produce strategic & logical frameworks

• Beginner’s workshop on program logic

18

Aspects of Influence – Strategic

• Early 2011 - R&E Audit & benchmarking - Developing guidelines

• Mid 2011 – R&E Strategic Agenda development - R&E Capacity building strategy

• Late 2011 – R&E Strategic agenda endorsed - Departmental governance

arrangement for all R&E resources

• 2012 – Implementation of strategic agenda

19

Influencing the Use of Evaluation Findings

Sue DoylePolicy and ResearchArts Victoria

• State government body which advises on and implements arts policy.

• Charged with making the arts available and accessible to all Victorians and with supporting and developing Victoria's artists and creative industries.

Arts Victoria

Rawcus Theatre Company – performers with and without disabilities

Survey of past evaluations

• How useful was the evaluation, how has it been used?• Were findings used or recommendations implemented? If

not, why?

Results Where recommendations were made, over half implemented Most useful in guiding program improvements &

communicating value of program, including lobbying for more resources

One issue with some recommendations (aside from lack of resources) is lack of understanding about operating environment of programs

l

Survey of past evaluations

• Ideas for improvement

Actionability & relevance of recommendations sometimes an issue

Could be addressed through evaluator providing findings & then facilitating discussion amongst key stakeholders (including senior executive and program staff) to develop recommendations or actions

• Greater ownership of evaluation process and findings by influencers and decision makers

• Still need political will and available resources

ExampleEvaluation of Castanet

• Castanet – community arts network• Castanet evaluated using more “action-research”

based approach• Dynamic relationship between findings and

program/initiative, increases likelihood and/or timeliness of addressing findings

• Anonymous feedback mechanism for network • Evaluator as advocate

Department of Primary Industries

Kirsty Fenton

“The systematic collection and use of information to improve decision making, demonstrate impact and performance of services to the public and key stakeholders, and enhance organisational learning”

If evaluations aren’t used, how can we expect them to be

influential?

While evaluation within FSV is implicitly utilisation-focussed, how evaluations could be used outside the immediate project is often not given the appropriate consideration when designing an evaluation.

Sometimes, evaluations are considered by staff as a compulsory project deliverable, but not as a vehicle for learning.

It is uncommon for evaluation findings and recommendations to be shared with other teams, or their findings utilised to inform policy and strategy.

Lack of formal process to share evaluation findingsLack of a central, searchable repository Project deliverable vs vehicle for learningVarying levels of quality & consistency in evaluation planning, implementation, reporting and utilisation.

Utilisation Scorecard

Evaluation

Findings

Improvement Inform Accountability

Own Project

Delivery

Other project

delivery

Investment

decisions

Strategy:

Change or confirm

Demonstrate

achievement of

outcomes and

value for $

New project

design &

development

Evaluation design &

implementation

(other projects)

X X X

Step 1: Understanding your project

1.1 Program logic

Step 2: Evaluation planning

2.1 Identify the evaluation audience

2.2 Define the purpose of the evaluation

2.3 Define the focus of the evaluation

2.4 Developing an evaluation plan

2.5 Key Evaluation Questions

2.6 Methods

2.7 Analysis

Step 3: Evaluation Implementation

3.1 Conducting the evaluation

3.2 Reflection

Step 4: Reporting

4.1 Reporting

4.2 Utilising evaluation findings and recommendations

4.2.1 Evaluation Communication Plan

4.2.2 Recommendation response and prioritisation

4.2.3 Evaluation summary

Templates

T1: Evaluation audiences

T2: Roles and responsibilities

T3: Evaluation work plan

T4: Evaluation communication plan

T5: Recommendation response and prioritisation

T6: Evaluation summary

Evaluation Summary

Captures important information, such as evaluation purpose, key findings, recommendations and reflections.

Summary sheetDetailsBranchKey Project CMI#MIS#Project team contactEvaluation team contactEvaluation titlePurpose of the evaluationCollaborators (eg. other PAG, Dept.)Completion dateContractor name (if applicable)

Summary sheet – cont.Project Outcomes-Key findings

Utilisation & recommendations-Have recommendations been developed and/or actioned? (if yes, add recc response & prioritisation template 5)

ReflectionEvaluation outcomesEvaluation process

Evaluation Outcomes•Are the findings relevant to other programs•Insights into:

• Strategy – are we doing the right thing?• Design – is our project/service design right?• Delivery – how well did we implement the project/service?

Improvements?

Evaluation Process•What went well? •What was frustrating? •What might you do differently next time? •3 things you’d suggest to someone else undertaking evaluation

Individual project level – increased utilisation by commissioner and project team, use of previous evaluations by other project teams

Key Projects – draw together findings and recommendations from multiple projects

Sector Service Evaluations

(Summit Workshop – KPM’s, branch directors, A&S, SET, project & evaluation managers – use also for FSV mini-conference, AFG evaluation & investment conferences)

Service Design branch

Strategy & Investment branch

- Analysis & Strategy – to inform policy & strategy

- Strategic Evaluation – meta-analysis & meta-evaluation (to inform ECB activities)

- KPM’s etc. for reporting & learning

Next Steps

Trialling summary sheet with project teams

Tailored training

Incorporation of evaluation into project management cycle

Influencing utilisation – take home messages

• Leadership & direction from the top, authorising environment • Support and ownership from decision makers and influencers

is crucial – participatory approaches encourage this• Ensure engagement with stakeholders throughout• Use authorising environment to your advantage• Processes & guides are only useful if people use them! • Bringing about change, especially using evaluation, is

challenging – So:• Sustained vigilance – just keep going!• Have a vision while knowing that it's a long journey - keep

pushing ahead to achieve the vision • When the right conditions for change come along - ride on it

and run with it