How to evaluate climate policy: Case of an evaluation-based audit of Finnish climate and energy strategy

2012 EEEN Forum

9-10 February, Leuven, Belgium

Paula Kivimaaa & Vivi Niemenmaab

a Senior Researcher, Finnish Environment Instituteb Principal Performance Auditor, National Audit Office of Finland

Pau

la K

ivim

aa, S

YK

E

2

● Challenges for evaluating climate policy

● Performance audit in comparison to policy evaluation

● Evaluation-based performance audit of Finnish climate and energy strategy

● Conclusions on challenges

10.

2.2

012

Contents

Pau

la K

ivim

aa, S

YK

E

3

● Uncertain and long-term cause-effect chains• Emergence of new scientific knowledge• Final impact measurement not possible

● Multiple other causes and policies influencing in desired and opposite directions

• E.g. economic fluctuations, policies supporting employee mobility, municipal interests

● Horizontal involvement of several administrative sectors and policy domains an important feature of climate policy

• How do we define climate policy?• Impacts on effectivess and evaluation

● Slowly changing practices• Despite awareness, it is difficult to change daily practices,

administrative processes, etc.• Marginal change or no change? Cause for change?

17.

10.

2011

Challenges for evaluating climate policy

Pau

la K

ivim

aa, S

YK

E

4

● There are not enough systematic evaluations of climate policy to support evaluation-based policy making (Haug et al. 2010)

● Existence of both technical-analytical and reflexive evaluations carried out by a number of actors (Huitema et al. 2011)

● “Several long-standing measures reported as climate policies were initially designed as responses to other problems” and “climate policy evaluations do not, in general, challenge dominant framings of the policy problem as one of market and state failure with regard to greenhouse gas emissions” (Haug et al. 2010)

● Evaluations should (1) address the complexity of climate policy making, (2) be reflexive in challenging policy goals and means, and (3) be participatory (Huitema et al. 2011)

17.

10.

2011

Findings from a climate policy meta-evaluation

Pau

la K

ivim

aa, S

YK

E

5

● ”the independent examination of the efficiency and effectivenss of government organizations, operations, or policies, with due regard to the economy’”(INTOSAI, 2010:11)

● Starts from the rationalistic side in that audits are carried out from the viewpoint of what states can do and should do better and relates to performance-based management

• Not so much concerned with e.g. rational social choice or future paths

● Often have a narrower focus than evaluation as a whole • focused on ex-post and have to be justified by direct or indirect

links to government economy

● Based on principles of good governance and decisions and goals set up in legislation (van der Knaap, 2011)

● Carried out by supreme audit institutions

17.

10.

2011

Performance audits

Pau

la K

ivim

aa, S

YK

E

6

● Carried out between August 2010 and November 2011 (circa 14 effective working months)

● Focus on the quality of policy preparation & the consistency, effectiveness and cost-efficiency of implementation

● Addressed complexity through multiple methods and data sources & addressed participation through a stakeholder questionnaire, interviews, and a focus group

• Was not reflexive in a sense that the main goals were taken as given from the EU level and did not address the framing of goals and measures

17.

10.

2011

Case of evaluation-based audit of Finnish climate and energy strategy

Pau

la K

ivim

aa, S

YK

E

7

The effectiveness model (source: Performance Audit Manual of the NAOF)

17.

10.

2011

Social needs and

objectives

Impacts

Outputs and outcomes

Managing and steering

activities

Organisation and

resources

Main focus of climate and energy strategy audit

Feedback to objective setting

Pau

la K

ivim

aa, S

YK

E

8

17.

10.

2011

● Was the preparation of the strategy of good quality?○ Was the preparation organised effectively, crossing administrative

sectors, transparently and involving different stakeholders?○ Was the information base versatile and sufficient, and was the

government economy comprehensively acknowledged?○ How were policy aims, instruments and their interrelations considered?○ How were the implementation and responsibilities planned?

● Has the implementation of the strategy strived for consistency, effectiveness and cost-efficiency from the perspective of climate change mitigation?

○ How does climate change mitigation shows in the preparation of government budgets?

○ Is implementation managed effectively and with sufficient resources?○ Has implementation been in line with the strategy statements?○ Has monitoring and evaluation been organised appropriately?

○ And more tentative evaluation of impacts and effectiveness

17.

10.

2011

Pau

la K

ivim

aa, S

YK

E

8

Audit questions

Pau

la K

ivim

aa, S

YK

E

9

17.

10.

2011

Audit criteria (more tentatively effectiveness)

Emission reduction & other legislation

based requirements (Kioto, EU)

Versatility and reliability of information

base

Long-term cost-

efficiency

Horisontality of

governance

Good governance

(transparency, participation)

Pau

la K

ivim

aa, S

YK

E

10

Methods and material

17.

10.

2011

Pre-study (3 months)•Key public policy documents•Focus group discussion with policy scientists•Interviews in key ministeries (3/5)• Audit questions and criteria

Main study (9 months)•Electronic stakeholder questionnaire (67 responses, 50% response rate)•Preparatory material (meeting minutes [43], background studies)•Sectoral strategies and background material•Interviews of government officials and experts (23/40) •Government budget proposals from 2008 to 2011•Statistics Finland

Ending the process•Internal quality control•Comment round in government administration•Final meeting of audit organisation•Publication of report (December 2011)

11

ener

gy s

ecur

ity

ener

gy e

fficien

cy

ener

gy s

aving

rene

wable

ener

gy

econ

omic

instru

men

ts

build

ing a

nd c

onst

ruct

ion

trans

port

com

mun

ity s

truct

ure

waste

man

agem

ent

agric

ultur

e

fore

stry

fore

sts

as s

inks

rese

arch

, tec

hnolo

gy &

inno

vatio

n

educ

ation

, adv

ice &

com

mun

icatio

n

stra

tegy

as

whole

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

3.38

3.042.82

3.53 3.51 3.51

3.04

2.622.86

2.54

3.11 3.033.23

3.00 3.09

Kaikkien vastausten keskiarvot

Preparation

Implementation

Effectiveness

General view of implementation in different sectors, questionnaire (5=excellent, 1=poor)

Pau

la K

ivim

aa, S

YK

E

12

17.

10.

2011

Example from the transport sectorNeed for transport

Transport mode

Vehicle efficiency

Fuel emissions

GHG-emission

s

Adapted from Monni & Raes (2008)

Policies (resources, measures)

Sectoral objective setting

(options, information

base)

13

Preparation

Implementation

Effectiveness

Measures influencing transport emissions

Area Supporting and hindering measures

Policy process - Sector specific target in climate and energy strategy (+)- Sectoral climate programme, high level network, monitoring (+)- Distribution of responsibility to agencies (+)

Amount of transport (reduction of need)

-Increase in fuel taxation (+) (M. of Finance, fiscal policy)-Joint planning of land use and transport (+) (horizontal, vertical)-Applications for intelligent transport (+)-Other communications policy (+/-)-Tax reductions of commuting costs (-) (M. of Finance)

Choice of transport mode

- Subsidies for public transport (+)- Development programme of public transport (+)- Strategy for walking and cycling (+)- Employee transport ticket (+)- Targetting of transport route investments (+/-)- Tax free parking benefit offered by employer (-) (M. of Finance)

Vehicle fuel efficiency

- Emission limits for vehicles (+)- CO2-based vehicle taxation (+) (M. of Finance)- Energy labelling system for vehicles (+)- Energy efficiency agreement for public and freight transport (+)-Training for economic driving(+)-Energy-efficiency criteria in public procurement(+)

Fuel emissions - Biofuels distribution obligation (+/-) (M. of Employment and Econ.)- Biofuel development programme (+) (MEE)- Subsidies for piloting and demonstration of transport biofuel production plants (+) (MEE)

14

Preparation

Implementation

Effectiveness

Effectiveness and emissions tentatively

Area Main factors increasing and decreasing emissions

Overall - The growth in transport has ’used up’ emissions reduction from improved vehicle efficiency and biofuels

Amount of transport (reduction of need)

- Tax reduction for commuting costs to support employee mobility (-)- Economic downturn (+)- New communication technology (+)

Choice of transport mode

-Budget funding for public and light transport exists (+)- Resources do not match the estimated need (-)

Vehicle fuel efficiency

- Emission limits to and labeling of new vehicles (+)- CO2-based vehicle taxation (+)- Lower than anticipated renewal of vehicle base (-)- Import of old vehicles (-)

Fuel emissions -- Distribution obligation for biofuels (+)

15

Preparation

Implementation

Effectiveness

Vehicle efficiency: kilometres per tonne of CO2

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

3000

3200

3400

3600

3800

4000

4200

4400

4600

4800

5000

5200

5400

Km/CO2 t

km

16

Preparation

Implementation

Effectiveness

Summary of implementation

• Significant new measures in renewable energy, transport and energy efficiency of buildings

• Major flaws: • Agriculture, community structure and land use (that affect possibilities

to reduce traffic and its emissions)• No real and innovative openings to promote micro- or consumer-

generation of renewable energy• Budget funds for the transport sector are small

• Monitoring and follow-up

• The official monitoring of strategy is weak, but fulfilled partly through monitoring of government programmes and to reporting to the UNFCCC and EU, which in turn are carried out well.

17

Valmistelu

Toimeenpano

Vaikuttavuus

Example from climate-based budgeting185 % increase in renewable and energy-efficiency, increases in transport and agriculture are small

2008 2009 2010 20110

50

100

150

200

250

300

R&D funding renewable energy energy (other)

energy-efficiency of buildings public transportation agri-environmental measures

climate policy in general

18

Preparation

Implementation

Effectiveness

Summary of effectiveness

• Renewable energy• Achieving 2020 goals seems challenging and dependent on forest

industry as well as the effectiveness of the feed-in-tariff system

• Agriculture’s N2O emissions• In present situation planned emission reduction appear unrealistic and

depend much on EU agricultural policy

• Transport CO2 emissions• So far growth in transport has exceeded the benefits from biofuels

and technological development.

• Community structure• Preconditions for effectiveness are missing outside the MoE:s

administrative sector

Requires further measures in many sectors and in energy efficiency

19

Recommendations of the audit

• Further increasing coordination between administrative sectors

• Improving participation and transparency

• Developing the evaluation of the cost-efficiency of measures and the evaluation of the impacts of the measures on government economy

• Drafting a climate budget

• Identifying subsidy elements that have contradictory effects

• Improving implementation in the weakest areas

• Clarifying the responsibilities of strategy monitoring and development of climate policy evaluation and reporting

Pau

la K

ivim

aa, S

YK

E

20

● It is relatively straight forward to evaluate the progress of policy processes and measures after target setting, the methodological challenges lie evaluating both cost-efficiency and effectiveness

● Evaluation of climate policy (even within sectoral emissions) complicated by

• Multiple policy domains, organisations, levels of governance, scope of emissions analysis (national reported vs. global) and lack of scientific precision regarding impacts

● The effectiveness of climate policies could be tentatively evaluated based on their potential for change

• Incremental vs. system-level change• Intervention logic and whether it is likely to work• Actor perceptions

17.

10.

2011

Conclusions

Pau

la K

ivim

aa, S

YK

E

21

Division of climate-based human resources in ministeries

17.

10.

2011

37 %

24 %

5 %

21 %

1 %

1 %

1 %

11 %

employment and economyagriculture and forestrytranport and communicationsenvironmentfinanceprime ministers officeforeign affairs