Evaluation of the Market Transformation Programme Final report

Evaluation of the Market Transformation Programme

Final report

April 2012

Evaluation of the Market

Transformation Programme

Final report


Final report

April 2012

Evaluation of the Market

Transformation Programme

Final report

Project

Directors:

Charles Michaelis &

David Kenington

Report Authors: Richard Carter

John Fawcett

Jennifer Hindson

Paula Owen

Additional Project

Team:

Katherine Netherwood

Debs Secker

Date: 19th April 2012


Final report

April 2012

Contents

GLOSSARY OF KEY TERMS .............................................................................................................................. 1

1 EXECUTIVE SUMMARY ............................................................................................................................. 5

1.1 INTRODUCTION.................................................................................................................................................. 5 1.2 OBJECTIVES OF THE RESEARCH .......................................................................................................................... 5 1.3 KEY FINDINGS ................................................................................................................................................... 6

1.3.1 Achieving positive outcomes .................................................................................................................... 6 1.3.2 How positive outcomes are achieved........................................................................................................ 6 1.3.3 How effectively the programme is managed and delivered ...................................................................... 7

2 INTRODUCTION ........................................................................................................................................... 9

2.1 BACKGROUND ................................................................................................................................................... 9 2.1.1 What is the Market Transformation Programme and what is it for? ........................................................ 9 2.1.2 What are the activities of MTP? ............................................................................................................... 9

2.2 OBJECTIVES OF THE RESEARCH ........................................................................................................................ 10

3 METHODOLOGY ........................................................................................................................................ 11

4 PROCESS EVALUATION .......................................................................................................................... 13

4.1 PROGRAMME LOGIC MAP ................................................................................................................................. 13 4.2 POST-2007 CONTRACT CHANGES AND RATIONALE ........................................................................................... 14 4.3 HAVE THE INTENDED BENEFITS BEEN ACHIEVED? ............................................................................................. 16 4.4 OTHER ISSUES ARISING .................................................................................................................................... 18 4.5 IMPLICATIONS ................................................................................................................................................. 19

5 POLICY IMPACT DATA USED ................................................................................................................ 20

5.1 MEPS / LABELLING ......................................................................................................................................... 20 5.1.1 ‘Policy vs. reference’ impact .................................................................................................................. 20 5.1.2 Lifetime impact ....................................................................................................................................... 21

5.2 LIMITATIONS OF THE CARBON BUDGETS AND MTP IA DATA: THE NECESSITY TO EXPLORE ‘POLICY A VS. POLICY

B’ IMPACT ............................................................................................................................................................ 22 5.2.1 Limitations of the Impact Assessment and carbon budgeting figures ..................................................... 22 5.2.2 Rationale for establishing a ‘policy A vs policy B’ scenario .................................................................. 24

5.3 OTHER POLICY IMPACTS .................................................................................................................................. 26 5.3.1 Voluntary agreements ............................................................................................................................ 26

5.4 GREEN PROCUREMENT .................................................................................................................................... 27 5.5 OTHER ............................................................................................................................................................ 27 5.6 WIDER BENEFITS ............................................................................................................................................. 27

5.6.1 European Benefits .................................................................................................................................. 27 5.6.2 Global impacts ....................................................................................................................................... 28 5.6.3 MTP work achieving non-carbon impacts .............................................................................................. 28

6 ATTRIBUTION: EVIDENCE OF MTP INFLUENCE ............................................................................. 29

6.1 STAKEHOLDER AWARENESS OF AND ENGAGEMENT WITH MTP. ........................................................................ 29 6.1.1 Perceptions of MTP ................................................................................................................................ 29

6.2 VERIFICATION OF MTP INFLUENCE ON MEPS AND LABELLING ........................................................................ 30 6.2.1 Phase 1 anticipated influence ................................................................................................................. 30 6.2.2 Acknowledged general UK / MTP influence .......................................................................................... 31 6.2.3 Barriers to identifying and achieving influence ..................................................................................... 33 6.2.4 Exploring the counterfactual .................................................................................................................. 34


Final report

April 2012

6.2.5 Specific influences at European level ..................................................................................................... 35 6.2.6 Refutation of influence and critiques of MTP data and the UK position ................................................ 36 6.2.7 Summary of influence on MEPS and labelling ....................................................................................... 36 6.2.8 Influences on non-priority products ....................................................................................................... 38

6.3 INFLUENCE ON PUBLIC (GREEN) PROCUREMENT .............................................................................................. 38 6.4 INFLUENCE ON VOLUNTARY AGREEMENTS ....................................................................................................... 39 6.5 INFLUENCE ON ENERGY STAR .......................................................................................................................... 40 6.6 OTHER UK-SPECIFIC POLICY INITIATIVES......................................................................................................... 41 6.7 ASSIGNING IMPACTS TO ATTRIBUTION BANDS .................................................................................................. 43 6.8 VALUE FOR MONEY ANALYSIS ......................................................................................................................... 45

6.8.1 Factors considered in the analysis ......................................................................................................... 45 6.8.2 MTP Value for money ............................................................................................................................ 46

7 CONCLUSIONS AND RECOMMENDATIONS ...................................................................................... 47

7.1 IS THERE VALUE IN THE OUTPUTS? ................................................................................................................... 47 7.2 OPPORTUNITIES FOR MORE EFFICIENT DELIVERY? ............................................................................................ 48

7.2.1 Why MTP? .............................................................................................................................................. 48 7.2.2 Improving efficiency within the current framework................................................................................ 49

7.3 ENHANCING PROGRAMME EFFECTIVENESS ....................................................................................................... 50 7.3.1 Retaining work in key areas of impact ................................................................................................... 50 7.3.2 Enhancing the impact of current outputs................................................................................................ 50 7.3.3 Future opportunities ............................................................................................................................... 51

8 APPENDIX A: FULL METHODOLOGY STATEMENT ........................................................................ 54

8.1 PHASE 1: DESK RESEARCH ............................................................................................................................... 54 8.2 PHASE 1: CONSORTIUM INTERVIEWS ................................................................................................................ 55 8.3 CONSORTIA WORKSHOP ................................................................................................................................... 55 8.4 PHASE 2: STAKEHOLDER INTERVIEWS .............................................................................................................. 56

8.4.1 Impacts explored .................................................................................................................................... 56 8.4.2 Stakeholder interviews conducted .......................................................................................................... 56 8.4.3 Challenges encountered in securing interviews in phase 2 .................................................................... 57

8.5 IMPACT METHODOLOGY .................................................................................................................................. 57 8.5.1 What impact data was required and why? ............................................................................................. 57 8.5.2 Sources of Impact Data .......................................................................................................................... 57 8.5.3 Overlap with other policies .................................................................................................................... 59 8.5.4 Impact data quality and limitations ........................................................................................................ 60

8.6 ATTRIBUTION METHODOLOGY ......................................................................................................................... 61 8.6.1 Stage 1: identifying potential areas of MTP influence ........................................................................... 61 8.6.2 Stage 2: verifying the claimed influences ............................................................................................... 61 8.6.3 Challenges in verifying MTP influence .................................................................................................. 62 8.6.4 Stage 3: Arriving at an assessment of attribution ................................................................................... 62 8.6.5 Stage 4: Integrating impact data with attribution data .......................................................................... 63


Final report

April 2012

1

Glossary of key terms

Terminology / acronym

Definition (in the context of this report)

IEA Implementing Agreement

for a Co-operating Programme

on Efficient Electrical End-Use

Equipment (4E)

An international collaboration between International Energy Agency

members to promote wider use of more energy-efficient electrical

equipment.

Attribution

An assessment of the extent to which policy outcomes and thus

improvements in energy performance can be ascribed to a

particular organisation or activity.

Attribution factors

Different levels at which the programme could have influenced

outcomes e.g. strong influence, some influence, little or no

influence.

Blocking minority

The number of votes required at EU Council votes to block a

decision. In EU 27, a blocking minority requires 91 votes of the 345

votes in the Council. This only applies when the ‗qualified majority‘

voting system is used.

Circulators

A circulator is a passive three- or four-port device, in which

microwave or radio frequency power entering any port is

transmitted to the next port in rotation (only). There are circulators

for LF, VHF, UHF, microwave frequencies and for light.

Cold appliances Appliances that use energy to cool e.g. refrigerators or freezers.

Commission policy leads

The Commission representative responsible for coordinating the

policy process – including assimilation of advice and information

from member states and other stakeholders - on a particular

product group within the Ecodesign of Energy-related products

Directive.

Compliance

Ensuring that product manufacturers are producing products that

meet the energy performance requirements of the relevant

regulations.

Confidence level

A statistical term assessing robustness of findings. In the context of

this report, this term refers to the extent to which we can be

confident of MTP claims of influencing particular policy outcomes

e.g. green (strong verification), amber (some verification), red

(little or no verification).

Consortium

Refers to the overall organisation of the MTP, with Defra managing

AEA who in turn contract and coordinate a range of sector and

product experts.

Consultation / negotiation

stages

Proposed EU legislation under the Ecodesign of Energy-related

products Directive must pass through a consultation stage (inviting

stakeholder views on the existing proposals) and a subsequent

negotiation stage (where the proposals are discussed, finalised and

voted upon by member states).

Cost effectiveness

Essentially an assessment of value for money, usually expressed as

a ratio (x benefit for every £1 spent). In the context of this report,

refers to assessment of the benefits delivered through MTP outputs

when compared to the costs of delivering the programme.

Counterfactual

Important in understanding the influence / impact of an

intervention, the counterfactual is the establishment – either

hypothetical or observed – of what would have happened without

that intervention.


Final report

April 2012

2

Domestic lighting Fixed lighting found in homes.

Enhanced Capital Allowance

(ECA) scheme

Capital allowances allow the costs of capital assets to be written off

against a business's taxable profits. They take the place of

depreciation in the commercial accounts, which is not allowed for

tax. These schemes have been introduced by government to

encourage investment in particular assets or by particular sorts of

businesses (in the context of this report, energy efficient products

and machinery).

Ecodesign of Energy-related

products Directive

Sometimes refereed to as simply the Ecodesign Directive, this set

of regulations aims to provide consistent EU-wide rules for

improving the environmental performance of energy related

products (ERPs) through eco-design. It prevents disparate national

legislations on the environmental performance of these products

from becoming obstacles to the intra-EU trade. It covers both

Energy-using products (EUPs), which use, generate, transfer or

measure energy and energy related products (ERPs) which do not

use energy but have an impact on energy and can therefore

contribute to saving energy, e.g. windows, insulation, taps etc.

Under the directive, performance standards are established for each

product group.

Electric motors

An electromechanical device that converts electrical energy into

mechanical energy. Electric motors are found in applications as

diverse as industrial fans, blowers and pumps, machine tools, and

household appliances.

Emergency Power Supply Unit

(EPSU)

A device which produces energy in case of failure of the primary

systems.

Energy Star

ENERGY STAR is a voluntary label developed by the US that

identifies the most energy efficient products on the market. Since

2000, the EU has had an agreement with the US to use the label for

office ICT equipment (such as computers, monitors and imaging

equipment). To be eligible for ENERGY STAR, products must meet

specified criteria including energy efficiency minimum standards.

European Commission

The executive body of the European Union, responsible for

proposing legislation, implementing decisions, upholding EU treaties

and the general day-to-day running of the Union.

Government Standard Briefing

Notes (GSBNs)

Notes containing evidence base information and details all

references and assumptions behind the figures in the evidence

base. These Briefing Notes are public consultation documents that

allow stakeholders to examine the data and assumptions behind the

proposed Government Standards and related projections.

Green Procurement

Taking environmental considerations into account within the

procurement process. For governments, introducing green

procurement means directing the combined spending power of

government towards environmentally friendly products and

services. Green Procurement encourages public sector buyers to

take account of environmental factors by setting out key

considerations for procuring different types of goods as well as

technical specifications and criteria that products should meet.

Impact Assessments

The principal outputs of the MTP programme to Defra. These

documents are product group-specific and provide UK government

with a briefing as to the current state of the market and the likely

impacts of introducing different requirements around product

energy performance.

Industry In the context of this report, refers collectively to manufacturers,

retailers and their trade associations / federations which consult


Final report

April 2012

3

upon and are affected by the MEPS and labelling legislation coming

out of the EU.

Labelling

A method of visually displaying the performance of products to

allow straightforward comparison or assessment of different

products. In the context of this report, a method of establishing a

mechanism for comparing the efficiency of energy-using products.

Lifetime impacts

For this report, the expected ongoing annualised financial /

environmental benefits arising as a result of policies to improve

product energy performance.

Member State One of the currently 27 countries that comprise the EU.

Minimum Energy Performance

Standards (MEPS)

Specifications containing a number of performance requirements for

an energy-using device, which effectively limits the maximum

amount of energy that may be consumed by a product in

performing a specified task. A MEPS is usually made mandatory by

government and generally requires use of a particular test

procedure that specifies how performance is measured.

National Measurement Office

(NMO)

An executive agency of the UK Department for Business, Innovation

and Skills (BIS), NMO is responsible for managing and developing

the National Measurement System (NMS) which is a network of

laboratories and processes that provide measurement standards

and calibration testing facilities. It maintains the measurement

infrastructure, represents the position of the UK internationally and

influences the development of standards.

Net benefits

Extent of benefits remaining after the costs of delivering those

benefits have been taken into account. In the context of MTP‘s

work, an example would be the overall financial / sales benefits

delivered to UK manufacturers by a drive for more energy efficient

products after the

Non-priority products

For the purposes of establishing MTP impact, non-priority products

are those which the consortium did not feel the MTP had influenced

much in terms of eventual MEPS / labelling policy outcomes.

Policy A vs Policy B scenario

The marginal / additional impact of the MTP-influenced policy

outcome compared to the policy outcome that would otherwise

have been realised.

Policy impact / benefit

In the context of this report, the annualised financial and

environmental benefits arising as a result of a particular energy

performance policy / standard being established for a product

group.

Policy outcome

The policy position arrived at, generally used in this report in

reference to the final MEPS / labelling legislation at EU level

following member state negotiations.

Policy vs Reference scenario

The impact of the MTP-influenced policy outcome compared to a

counterfactual / (‗reference‘) scenario in which no policy is

introduced.

Product area / group The groups into which products governed by EU legislation are split.

Super-Efficient Equipment and

Appliance Deployment (SEAD)

Similar to the 4E, the SEAD initiative is a voluntary scheme with

government members which seeks to transform the global market

for efficient equipment and appliances—reducing energy

consumption while simultaneously saving money for consumers.

SEUP Benefits document

A document produced internally within the MTP, which estimated

the extent to which UK benefits derived from EU MEPS / labelling

policies could be attributed to MTP activities. This formed the basis

of the impacts that this evaluation sought to verify.

Simple Set Top Boxes (STBs) A device that generally contains a tuner and connects to a TV,

turning the signal into content which is then displayed. Set-top


Final report

April 2012

4

boxes are used in cable and satellite TV systems to transform the

signal from the cable or satellite to a form that can be used by the

television set or other receiver. It also enhances the quality of

signal.

Stakeholders In the context of this report, refers to any individuals involved in

MTP consortium work i.e. Defra representatives, AEA PAs, or PAMs.

Standby Refers to the electric power consumed by electronic and electrical

appliances while they are switched off or in a standby mode.

Subcontractors

The experts / PAMs contracted by AEA to provide product specific

data and expertise to enable development of the Impact

Assessments and other MTP outputs.

Tertiary lighting Apart from fixed lighting, the other lighting found in homes or

businesses e.g. lamps.

Voluntary agreements In this context, agreements – generally between businesses - that

achieve higher than required environmental outcomes.

Wet appliances Appliances that use water to perform their function e.g. washing

machines or dishwashers.


Final report

April 2012

5

1 Executive Summary

1.1 Introduction

The Market Transformation Programme is a consortium of expert contractors managed by

the Department for Environment, Food and Rural Affairs (Defra) and coordinated by the

lead consortium partner, AEA Technology (working alongside Consumer Research

Associates).

The aim of the Market Transformation Programme (MTP) is to drive improvements in

energy and environmental performance of energy using products, through the provision of

a robust evidence base on the current and likely future performance of energy using

products for policy makers in the UK, the European Commission, industry, and global

projects such as Energy Star and 4E.

The main outcomes that the programme aims to support are a real reduction in emissions

(e.g. CO2) and energy use through:

New and refined policies and standards;

Better informed policy decisions (through provision of robust information);

Innovation and improved product design.

1.2 Objectives of the research

Defra‘s contract with AEA Technology to manage the MTP programme is due to end at the

end of October 2012; this presented an ideal opportunity to explore the outcomes and

impacts of the programme through its activities conducted between 2007 and 2011,

assessing the overall contribution of the programme to reducing energy use.

The primary focus of the evaluation was therefore to quantify the impacts influenced by

MTP and to therefore provide an estimate of value for money.

It was expected that the evaluation would inform Defra‘s decisions about:

The value of retaining an evidence base.

How the focus and management of the MTP contract post-2011 could be refined or

re-defined.

This evaluation of the MTP programme has therefore sought to establish:

1. Whether, and to what extent, the programme influences positive outcomes, with a

primary – but not exclusive – focus upon reduction in CO2 emissions.

2. On the basis of the impact attributed to MTP activities, whether or not value for

money has been achieved in securing outcomes.

3. How – and how effectively - the programme influences positive outcomes.

4. How effectively the programme is managed and how effective the current MTP

process has been.


Final report

April 2012

6

1.3 Key findings

1.3.1 Achieving positive outcomes

The evaluation has demonstrated that the programme does influence positive outcomes

and – in doing so – has achieved very strong value for money. Using the total additional

impact of the MTP-influenced policy outcome (when compared to the policy outcome that

would otherwise have been achieved), the impact and value for money of the programme

are estimated as follows:

Taking into account the lifetime benefits of policy outcomes which MTP outputs

have had a strong influence upon, the total programme benefits derived from

2007-11 activities are at least £348m and 1.61MtCO2. Compared with a total

contract spend of £12.2m, this equates to influenced savings of

approximately £28.50 and 0.13tCO2 per £1 spent.

If lifetime savings upon which Defra – underpinned by MTP – have had some

influence are also taken into account, this equates to influenced savings of at least

£954m and 5.38MtCO2. Compared with a total contract spend of £12.2m,

this equates to influenced savings of approximately £78.20 and 0.44tCO2

per £1 spent.

With a number of Ecodesign and Energy Labelling implementing measures still to be

negotiated, ongoing development of UK procurement standards and voluntary agreements,

and an important role for programmes such as Energy Star and 4E, there is also ongoing

potential for the benefits derived from 2007-11 activities to continue to be realised.

1.3.2 How positive outcomes are achieved

The evaluation shows that development and possession of the types of output that MTP

produces have:

Given the UK a prominent role in pre-consultation scoping and a prominent

position at the European negotiating table;

Ensured a level of (almost automatic) support from some Member States for the

UK position;

Provided an important counter-balance to less progressive national / industry

interests;

Helped the UK to maintain some degree of control over the direction of policies

that could otherwise carry detrimental effects nationally;

Built good relations and reputation with stakeholders both domestically and

internationally;

Meant the UK does not have to use more political means of negotiating to a

desired outcome;

Helped to embed more effective monitoring and compliance.

For specific policy outcomes, there is a level of uncertainty as to the counterfactual – i.e.

what would have happened without MTP outputs – but overall it is clear that some policy


Final report

April 2012

7

outcomes might have been very different without the UK taking an evidence-based

position.

A small number of stakeholders implied that other Member States can get by in

negotiations without the same support and that the UK – with a large number of votes –

could probably still enjoy some influence regardless of MTP. However:

Most respondents felt that a Member State would struggle to play an active role in

shaping policy without technical evidence bases and data.

Without MTP work, the UK could be less clear on which policy outcome would be in

the national / environmental interest and therefore what position to support.

The benefits of MTP outputs are not only realised in Commission negotiations, but

across a range of national and international initiatives and programmes.

1.3.3 How effectively the programme is managed and delivered

Whilst most respondents agreed that the type of outputs produced by MTP are beneficial,

most other Member States do not have a separate government-funded programme to

deliver these. The alternatives – as demonstrated by other Member States – were to have

advisors from within government (often energy / environmental departments or agencies)

or industry.

Alternative delivery methods – e.g. Defra directly managing experts - are options that

Defra may wish to consider. However, it is also important to highlight that the research

identified a number of arguments for continuing to source the required data through MTP

e.g. consistency, impartiality and building upon lessons learned.

Overall, whilst this evaluation indicates that there is value in both alternative

arrangements and retaining the current set up, it has certainly highlighted a number of

ways in which the current MTP process could work more efficiently and effectively:

Defra still need the outputs from MTP, but how much of the process of obtaining

them is now necessary?

There are recognised to be occasions where the UK has not been able to influence

policy outcomes. This suggests a need for more strategic assessment of whether or

not, for a particular product group or policy:

a. The effects of changes to a product group upon the UK are of enough

significance to warrant full investigation and analysis of data.

b. The eventual policy outcome is likely to be affected by the UK position.

c. The eventual policy outcome is likely to be influenced by possession of robust

data.

It should be borne in mind that where there is at least some impact of a policy /

product group upon the UK, there will still be a degree of analysis required for

carbon budgeting purposes. However, the resources devoted should be

proportionate rather than uniform.

Could the consortium be organised in line with Defra team i.e. on a product by

product basis?


Final report

April 2012

8

One of the key issues cited by MTP for turnaround of required outputs was the

management and availability of subcontractors. This is an issue common to

subcontracting that perhaps should have been anticipated. Solutions include a

requirement in the contracts of independent experts to commit to delivering work

or making themselves available, or an expansion of the panel in order to minimise

the risk of expertise being unavailable for a particular product group or policy area.

A separate study is currently being undertaken to test the feasibility of the second

option.

Retain resources for workstreams where MTP impact is clear e.g. Eco-Design,

international programmes and future studies.

Although there is generally agreed to be no modelling system currently better

placed to form an assessment of the impacts of the policies under consideration,

there is a need for enhancements to be made to the quality of the assumptions

and modelling used in the MTP evidence base. It could be that more frequent

revisions to the IAs may be necessary to update the evidence bases used for

modelling, particularly in fast-changing markets such as consumer electronics and

ICT. Clear markings of the updates and the significant changes that have been

made is also recommended.

Furthermore, there are a number of policies for which the outcomes and benefits

at a UK level are unclear; there would be benefit in seeking to evaluate these

policies and quantify their impacts more accurately.


Final report

April 2012

9

2 Introduction

2.1 Background

2.1.1 What is the Market Transformation Programme and what is it for?

The Market Transformation Programme is a consortium of expert contractors managed by

the Department for Environment, Food and Rural Affairs (Defra) and coordinated by the

lead consortium partner, AEA Technology (working alongside Consumer Research

Associates). There are also a number of additional technical advisers to the programme,

including the Building Research Establishment (BRE) and Intertek Research and Testing

Centre.

The aim of the Market Transformation Programme (MTP) is to drive improvements in

energy and environmental performance of energy using products, through the provision of

a robust evidence base on the current and likely future performance of energy using

products for policy makers in the UK, the European Commission, industry, and global

projects such as Energy Star and 4E.

The main outcomes that the programme aims to support are a real reduction in emissions

(e.g. CO2) and energy use through:

New and refined policies and standards;

Better informed policy decisions (through provision of robust information);

Innovation and improved product design.

2.1.2 What are the activities of MTP?

MTP is directly involved in the following activities:

Managing the energy using product evidence base, a key element of which is the

producing market intelligence and indicative standards for energy using products;

Assisting Defra (in negotiations with other member states) and the European

Commission in developing the eco-design of Energy Related Products (ErP)

Directive, which sets standards for the minimum energy performance of products

sold in the EU;

Contributing to international collaborations around standards and labelling e.g.

Energy Star and the International Energy Agency implementing agreement on

Efficient Electrical End-use Equipment (IEA 4E);

Influencing UK Government policy development;

Providing accurate assessments of policy impact for UK Carbon Budgets

accounting;

Up to 2009-10 - when this role was transferred to the National Measurement Office

(NMO) – assisting with compliance testing and the formulation of compliance

testing methods and standards.

MTP supports and drives forward these activities through its programme of work

(agreed with Defra).


Final report

April 2012

10

2.2 Objectives of the research

The 2007-11 Defra contract with AEA Technology to manage the MTP programme is due to

end at the end of October 2012; this presented an ideal opportunity to explore the

outcomes and impacts of the programme through its activities conducted between 2007

and 2011, assessing the overall contribution of the programme to reducing energy use.

The primary focus of the evaluation was therefore to quantify the impacts influenced by

MTP and to therefore provide an estimate of value for money.

The evaluation was not expected to be an assessment of the rationale for the programme,

or the policies it assists with, particularly on the basis that Defra are looking to re-tender

the contract. However, Defra did see the contract renewal as a logical point at which to

test the focus, structure and delivery of the contract throughout the 2007-11 contract

period. In addition, the assessment of the impact of the programme inevitably raises

questions around the value of particular activities and / or how impacts could be enhanced.

It is expected the evaluation will inform Defra‘s decisions about:

The value of retaining an evidence base.

How the focus and management of the MTP contract post-2011 could be refined or

re-defined.

This evaluation of the MTP programme has therefore sought to establish:

1. Whether, and to what extent, the programme influences positive outcomes, with a

primary – but not exclusive – focus upon reduction in CO2 emissions.

2. On the basis of the impact attributed to MTP activities, whether or not value for

money has been achieved in securing outcomes.

3. How – and how effectively - the programme influences positive outcomes.

4. How effectively the programme is managed and how effective the current MTP

process has been.


Final report

April 2012

11

3 Methodology

The section below provides a short overview of the research elements undertaken, the

rationale for these, and a brief summary of the approach to establishing programme

impact and attribution of that impact. A much more detailed methodology – and the

rationale for decisions around use of sources of impact data - can be found in

appendix A to this report.

The evaluation was split into two phases:

Phase 1 comprised desk research and interviews with MTP consortium

representatives. The purpose of this phase was to:

a. Ascertain the resources used to deliver the programme, and how these are

distributed.

b. Identify activities undertaken by the programme and outputs and impacts

arising from these.

c. Provide feedback on the programme logic, structure and day-to-day working in

order to enable a full process evaluation.

d. Enable a short-listing of product groups and policy outcomes where MTP

influence was felt to be strong. This was done on the basis that MTP activities

were numerous and wide-ranging, therefore it would be much more effective

for Defra and MTP to highlight areas where they believed there had been MTP

impact (which could then be verified) rather than seeking to gain specific

impacts from a large number of wider stakeholders.

There then followed an interim workshop with consortium representatives at

which phase 1 findings were presented and discussed, and attendees were able to

input further into any research questions pertaining to phase 1 objectives.

Phase 2 comprised interviews with 32 stakeholders, including Defra and AEA

product leads as well as representatives of the European Commission, member

states, industry and Energy Star programme. The purpose of these interviews was

to verify the influence of MTP upon the policy outcomes shortlisted in phase 1.

The attribution factors which would ascertain the extent of MTP influence upon

policy outcomes were developed into a two-factor assessment:

1. An assessment of how important MTP input was to the policy outcome realised,

on a scale of strong influence – some influence – weak influence – no

influence;

2. An assessment of how much confidence can be placed in this importance

rating, based upon such factors as number of respondents verifying, the source

of those verifying (were they authoritative / potentially biased?) and the level

of disagreement between respondents on the extent of MTP influence upon a

particular outcome. This was rated through a traffic light system of:

Green – good level of confidence

Amber – some confidence

Red – low confidence


Final report

April 2012

12

To enable an assessment of impact influenced by MTP, the attribution factors had

to be applied to projections of the impact that influenced policies would have in

terms of UK CO2 and cost savings. These impact figures were drawn from the latest

carbon budget figures for policy impacts in each product group. These figures are

derived from the MTP Impact Assessments and reflected (in an aggregated form)

in the DECC Carbon Plan.

The attribution and impact data were integrated to produce categories of impact

i.e. impact from policy outcomes which MTP had a strong influence upon, impact

from policy outcomes which MTP had some influence upon etc.

The policy IA figures are calculated by taking the difference in impact between the

policy outcome and a scenario in which there is no policy at all. We have therefore

also reported impact in a different way to provide a more accurate representation

of the likely impact MTP directly influenced i.e. taking the difference in impact

between the actual policy outcome and the policy outcome that would have arisen

without MTP input. The IA and our approach are described respectively as the

‗policy vs. reference‘ impact and the ‗policy A vs. policy B‘ impact. The latter was

calculated by the following process:

- In MTPs internal ‗SEUP Benefits‘ document1, an attempt was made to estimate

the extent to which the impacts achieved by MEPS policy outcomes for each

product group were attributable to Defra / MTP contributions (looked at

another way, what impacts from policy for this product group would not have

been achieved). This effectively points to a policy B / counterfactual policy

outcome that would have occurred without Defra / MTP input. Due to the

qualitative nature of this exercise, these estimates were reported as % ranges

i.e. Defra / MTP can claim between x% and y% of the impacts derived from the

policy outcome for product group z.

- The mid-point of this % range was applied to the latest carbon budget impact

figures for each product group to provide an estimate of the additional impact

of the MTP-influenced policy compared to the policy that would otherwise have

arisen i.e. ‗Policy A vs Policy B‘.

Both sets of final attributed impacts were then compared to MTP budget data to

enable ‗value-for-money‘ assessments.

1 This document was produced by Andy Cotterill and Davide Minotti as an attempt to quantify the

impact MTP has achieved through its work. The impact figures presented in the document are based

on the impact assessments used in MTP‘s modelling work.


Final report

April 2012

13

4 Process evaluation

This section uses data from the desk research, phase 1 consortia interviews and consortia

workshop to enable an assessment of the key successes and challenges of programme

delivery between 2007 and 2011. This feeds into the overall assessment of programme

appropriateness and effectiveness in section 7.

4.1 Programme logic map

The diagram below shows a logic map for the way in which the MTP programme results in

positive impacts for the UK in terms of reduced carbon emissions and financial savings.

Programme Inputs: Defra funding,

AEA and subcontractor expertise

Programme Workstreams:

Programme Management, EuP

Directive, Government Standards,

Evidence Base, International, Future

Studies, OGD and Ad-hoc, Specialist

Advice

Programme Impacts: Reduction

in energy usage, reduced CO2

emissions, financial savings.

Policies:

MEPs

and

labelling

Voluntary

Agreements

Energy

Star

Green

Procurement

Programme Outputs:

MTP

Evidence

Base

GSBNs Direct contact

with industry

Direct liaison

with

European

Commission

Ensure

s

com

pliance

Ensure

s s

ett

ing

of re

asonable

sta

ndard

s

Incre

ases

effectiveness o

f

policie

s


Final report

April 2012

14

4.2 Post-2007 contract changes and rationale

The MTP programme has operated as a discrete programme since 1998. Since 2002, a

panel of experts, including ECI Oxford University academics and AEA Technology staff,

have worked with Defra to develop an evidence base which would allow Defra to influence

policy negotiations.

In 2007, the current MTP contract was tendered. This contract renewal contained three key

changes:

1. A change of structure to encourage the use of a wide panel of product and

energy experts; this was a stipulation explicitly introduced as part of the new

contract specifications. Guidelines were set that limited funding for programme

management to 20% of the total budget, with the remaining 80% ring-fenced for

procurement of outputs from the expert panel.

The benefits of this approach were argued to be a wider engagement and drawing

in of sector-specific and specialist expertise, a more open contract, potentially new

approaches / data and less reliance upon one contractor. The purpose of

continuing to outsource the management of the programme and coordination of

the panel was that this would ensure a consistent formatting of outputs,

streamlined reporting to Defra, and ensure that there were clear lines of

responsibility for managing and transforming Defra requests into outputs.

The programme management element was won by AEA Technology, who had also

been the previous lead contractor. Generally 10-20% of MTP analysis and reporting

is done by in-house AEA staff, and 80-90% is done by external subcontractors.

The outputs generally take the form of a short report with both high-level findings

and some detail of the analysis conducted. MTP managers will then communicate

this work back to Defra. Defra will then use the outputs for influencing EU policy

making and standard setting and reporting on UK carbon budgets progress.

2. A continued shift of focus from UK to European policy: this change occurred

during the course of the 2007-12 contract. Initially, MTP focused upon

development of the evidence base, modelling and UK policy such as the

Government Standards work. This came about as a result of the Energy White

Paper in which developing UK product policy was seen a key approach in helping to

meet 2050 carbon reduction targets.

Over time, with the development of the EU Ecodesign of Energy Using Products

(lately re-cast as Energy Related Products) Directive in particular product groups,

the programme focus has evolved towards responding to policy and standard

setting by the European Commission, in the form of minimum energy performance

standards, labelling and other interventions. This was intended to ensure that MTP

outputs could be relevant and influential at the European level, though the outputs

are also required for carbon budget accounting purposes.


Final report

April 2012

15

3. A move from a more collaborative arrangement to one of Defra as clear

lead, especially on policy: with regards to MTP programme structure, the

number of Defra policy staff has increased over the contract from three to a team

of seven. In line with this, the extent to which the MTP consortium gets involved

with direct policy development and influencing has decreased. Defra now leads on

policy work with the European Commission, whereas before 2007 there was more

of a partnership approach to this between Defra and the consortium members.

The rationale for this change was that now that the Ecodesign Directive is being

progressed, there are potential risks for the UK government of being represented

by – and having UK policy outcomes negotiated by - independent consultants.

Defra policy leads now focus on specific product groups whilst the MTP consortium

now focuses more on producing the evidence base and modelling for two main

purposes:

Modelling and impact assessments (IAs) for Defra to use in EU negotiations

and carbon budget accounting.

IAs for economists, industry and other organisations e.g. other government-

funded programmes or agencies.

Defra continues to provide information to MTP on the product areas and analysis

required in response to EU negotiations. Defra may ask for MTP to conduct work at

a number of different points in the European Commission‘s legislative process,

including:

Pre-consultation (preparatory studies).

At the consultation stage, when a product area is being discussed, but before a

policy has been drafted.

At formal meetings after a policy has been drafted, where Defra would seek to

use the MTP evidence base to influence other member states.


Final report

April 2012

16

4.3 Have the intended benefits been achieved?

The table below sets out the benefits arising from each of the three principal contract changes outlined above, and any issues arising from these, to

provide an overall assessment of how effectively the changes have delivered the benefits anticipated:

Programme

element Benefits observed Issues arising

Moving to the 80% -

20% delivery and

management split.

The programme is drawing in more individual expert resource and

knowledge through the sub-contracting arrangement.

This means more organisations are potentially able to contribute to

outputs which can deliver benefits through identifying new data

sources or analysis approaches.

For Defra, this represents an expansion of contacts and knowledge

base.

To some extent, the apparent diversity of the expertise drawn

upon in the new process is misleading, as a proportion of the

current panel of individual experts are ex-AEA staff who left the

organisation post-2007 and are now freelance.

AEA and Defra recognised that the change had led to a loss of in-

house management and expert skills, thus impacting upon quality

of outputs and coordination where inexperienced individuals were

taking on these management roles.

In addition, some AEA staff argued that the requirement to

outsource delivery of work-streams previously delivered in-house

has led to inefficiency and delay in turning around work as time is

needed for these external resources to be commissioned and work

sub-contracted i.e. experts were not available when required

because they have other work commitments which are now outside

of the contract and which AEA cannot manage / control. The

programme work plans did show certain outputs being delayed and

delivered beyond schedule, yet it is not clear how far this was the

case pre-2007.

Concern about delays and inefficiency in the process has led to

occasional deviation from the intended process i.e. one respondent

said that on some occasions the contract procedure is being

bypassed and the sub-contracted expert is reporting directly to

Defra rather than passing outputs through AEA.


Final report

April 2012

17

A greater focus upon

European level policy

Assuming that MTP‘s core purpose is to provide data to inform

policy positions and outcomes, this change was essential to the

ongoing value and relevance of the programme.

It has ensured that Defra can bring useful and persuasive evidence

bases to the negotiating table to support their policy position and

influence the Commission. Section 6 explores and demonstrates

the influence that the UK – through use of MTP data – has enjoyed

in setting product policy.

Notice given for draft regulation by the EU can be very short,

giving little time for MTP to produce impact work on different

product policy scenarios; the inference from some respondents

was that this could impact upon quality and accuracy, or

turnaround times.

With support from Defra, MTP have attempted to organise

preparatory work to try to pre-empt which product groups might

be covered in legislation, though some requirements have still

necessitated long working hours for consortium staff. Attempting

to pre-empt requirements had also proved difficult due to the

changing nature of European policy formation. Whilst the

consortium felt that they had responded appropriately wherever

possible, this had been disruptive to normal, planned work and

resulted in working very long hours, with again the risk of

compromised quality.

Defra taking the lead

on policy

negotiations

This has ensured that the UK government are representing the UK

in all negotiations, ensuring impartiality and accountability.

However, this has not meant that MTP cannot be directly involved

in the process. Some experts are still being seconded to work with

the Commission on particular product groups and policies in initial

stages of development.

For some in AEA, the change in relationship between Defra and

MTP from one of a ‗partnership‘ (in which MTP‘s experts were

involved in shaping policy) to MTP becoming more of a service

delivery agent was seen to have led to less direct collaboration

between MTP and other EU and UK stakeholders and so less

strategic understanding from within MTP. This was felt by AEA

respondents to have led both to a stifling of innovation within MTP

– as they are reactive to Defra priorities - and a limit to the

additional impacts MTP could have through broader engagement.

One respondent strongly felt that the MTP has been “rather left

behind” as the policy agenda has moved on from that when the

programme was developed. They felt that this was at least in part

due to a failure of AEA to accept their altered role from leading the

thinking.


Final report

April 2012

18

4.4 Other issues arising

Although not an inherent result of the changing programme process or structure, certain

aspects of the contract and programme delivery were highlighted by respondents in phase

1 as having created issues:

Some AEA respondents stated that quite high staff turnover at Defra (as well as

product group complexities) has created inefficiencies, in that it takes time to bring

new Defra staff up to speed with the background to – and technical issues

surrounding – a particular product group area. Whilst AEA have worked with new

Defra staff to bring them up to speed with particular product groups and the

policies affecting them, this is an ongoing requirement not explicitly accounted for

in the contract.

Closer Defra scrutiny of the assumptions and modelling used to produce the IA

documents have identified issues, in particular the economic basis for assumptions

about changes in product markets was cited as an area where there are problems

with accuracy. This has sometimes led to the need for Defra economists to re-work

figures or request re-calculation. It was perceived by several respondents that

there was a lack of economics expertise within AEA and the subcontractor panel;

despite Defra providing some training on this issue to MTP, it was felt by one

respondent that the wrong individuals had been sent to this and that the learnings

had not been used fully. It should be noted that Defra have since seconded an

economics expert to AEA.

It was felt by some AEA respondents that there could have been wider

communication of MTP outputs across industry and other national stakeholders.

That this was not done was felt to be largely due to a lower priority being placed

on communication of outputs to a wider audience, as well as a lack of specialist

marketing expertise. Examples were cited where MTP outputs had been used and

communicated by others (e.g. The Energy Saving Trust) more than by the

programme itself. Whilst MTP not promoting their own works is not necessarily an

issue, where others (such as EST) do not do it, engagement with wider

stakeholders could be more difficult.

One respondent felt that the work on the programme is not well targeted,

providing too much theoretical evidence which is “miles away from policies” and

that the individuals in AEA providing advice don‘t have a good policy

understanding2. They argued that Defra being tied into the 2007-11 contract and

not being able to renegotiate has been problematic and led to situations where AEA

would cite contract limitations as a barrier to taking on particular workstreams.

They felt that AEA has “a strong institutional memory and talks in 2005-speak”.

They did accept that Defra had been tough on the contract costs but also felt that

AEA could have been more creative in suggesting savings from the core tasks.

2 Examples cited were:

a. Work on potential tightening of government buying standards, which while theoretically

possible, went against three key constraints – the government‘s commitment to no

regulations, George Osborne‘s position on climate change and the need for growth.

b. A desire for fiscal measures as well as regulations and labels which the respondent felt was

bad policy making as there is loss of value in multiple measures.


Final report

April 2012

19

A technical challenge highlighted is that as the programme has evolved, there has

been an increase in the complexity of the modelling system and the amount of

data it holds. The consortia interviews conducted indicate that for at least some

product areas, the system of Excel spreadsheets used to model different policy

scenarios has become inefficient to maintain and update. This was seen as a

potential risk to the modelling process if the demands on this system were to

further increase. One respondent described the MTP model as “a mess”, citing it as

one of the reasons why data provision from the programme is so expensive and

subject to delays. AEA were paid an extra £150k through the 2010-11 contract to

revamp the model but it is not clear how well some of the issues with it have been

resolved.

4.5 Implications

Overall, the process established for the 2007-11 MTP contract has still enabled the desired

outputs to be delivered and – as shown in section 6 – outcomes to be influenced.

In addition, despite the current contract structure and process being cited as responsible

for a number of issues, these should all be viewed in the context of:

a. The benefits obtained by Defra (outlined in the table in section 4.3) of the changes

to the contract;

b. Loss of influence and contract involvement for AEA; which always had the potential

to result in negative perceptions of some of the changes from AEA‘s point of view.

Some of the issues cited e.g. loss of proximity of MTP to European policy makers,

may not be seen by Defra as an issue at all.

c. Attempts have been made to address some of the issues that have arisen e.g.

training on economics to MTP policy advisors.

However, there is a suggestion that outputs produced are sometimes of questionable

quality and accuracy. Also, what is surprising about some of the issues raised is that some

issues arising from the 2007-11 changes could have been anticipated and processes /

procedures embedded to mitigate these. For example, the issues highlighted by AEA

around supplier delays, commissioning bureaucracy and lack of subcontractor control are

common to any subcontracting relationship.

Section 7 of the report considers the issues raised and provides some recommendations on

the ways in which these could be resolved.


Final report

April 2012

20

5 Policy Impact data used

This section sets out the impacts of policies which MTP had potential to influence.

Rationale for the sources of these figures, their limitations and overlaps with other policies

can be found in the final sub-section of this section of the report, in the methodology

appendix and in the conclusions of this report.

As most robust monetary and carbon impact figures available for MTP‘s work, DECC-

approved policy impact data was used as the basis for programme impacts.

In general, impacts associated with the implementation of the EU Ecodesign and Energy

Labelling Directives have been drawn from the ‗SEUP Benefits‘ analysis and reported on a

product group by product group basis. Whilst some estimated impacts were found for other

policies (such as Green Procurement and voluntary agreements), these were generally EU-

wide, rather than UK-wide impacts, and often lacked the analytical detail of the impact

assessments which make up the ‗SEUP Benefits‘ analysis.

5.1 MEPS / labelling

5.1.1 ‘Policy vs. reference’ impact

Since the MTP Impact assessments view minimum energy performance standards (MEPs)

and energy labelling together, we have used the combined impacts of MEPs and labelling

for the relevant product groups when assessing impact. It was also felt that phase 2

respondents would be better able to assess MTP contribution to the suite of policies in each

product group rather than separating them out.

The table below shows the average annual net financial benefit to the UK of each product

group as set out in the latest departmental figures. These figures – drawn from the latest

carbon budget figures - are based upon (but updated from) MTP impact assessments.

The savings presented in the table below are savings net of the reference scenario - a

projection of what is likely to happen to the energy consumption of each product area if no

new policies were implemented i.e. a reference scenario.

The effect of three different scenarios on the product markets is modelled in MTP‘s work:

The Reference Scenario is a projection of what is likely to happen to the energy

consumption of each product area if no new policies are implemented; for

clarification, this scenario does include all policies that have been implemented by

a certain point.

The Policy Scenario is a projection of what will happen if a defined set of

additional product-specific and related cross-cutting policies are implemented fully

and in a timely manner;

The Best Available Technology (BAT) Scenario is a hypothetical projection of

impact if the best available technologies on the market (current and future) are


Final report

April 2012

21

adopted. The scenario tends to assume implementation at the earliest possible

date and is unlikely to occur in reality, but gives an indication of potential savings.

The table below shows the average annual net financial and CO2 emissions benefit to the

UK of each product group as set out in the carbon budget figures (themselves derived from

the MTP IAs):

Product groups Average annual UK

net benefit from policies

Lifetime UK net benefit

from policies

Net annual CO2

savings predicted to be achieved in 2020 as a result of policies (Mt

CO2)

Standby £288m £3,165m 1.80

ESPUs £11m £125m 0.10

Electric Motors £92m £1,007m 1.30

Circulators -£11m -£119m 0.30

Cold appliances and labelling £44m £487m 0.60

TVs and labelling £64m £707m 0.70

Wet appliances and labelling £14m £149m 0.20

Domestic Lighting £124m £1,368m 0.50

Tertiary Lighting £214m £2,349m 1.30

Simple STBs £82m £899m 0.40

Total £922m £10,137m 7.10

5.1.2 Lifetime impact

The table below shows an estimate of the likely annual CO2 savings as a result of policies

in each product from 2010 to 2020.

This is included to illustrate the likely savings that would result from policies in each area

over time. These estimates have been produced by applying the net CO2 saving in 2020

produced in the departmental figures (itself derived from the published IAs and DECC

Carbon Plan) to the profile of net CO2 savings reported in each impact assessment. In

most cases, the impact assessments assume an s-shaped take-up curve between 2010

and 2020, resulting in higher net savings figures in 2020 than in the preceding years.

A take-up curve of this shape seems sensible intuitively, as it will take some time in the

early years of a policy for it to be widely adopted, and take-up will gradually plateau as the

market becomes entirely compliant. However, a critical issue is that the curve does not

seem to take into account the inevitable ramp down of the additional impact of the policy

as the market / general environment could have progressed to delivering these impacts

eventually3. Although this ‗ramp down‘ effect was adjusted for in carbon budgeting work,

this does not seem to have resulted in the declining net annual impacts that might be

expected.

3 In recent IAs this has been picked up by Defra economists and a revised approach is being

developed in response to this.


Final report

April 2012

22

Product Area

Estimate of Net CO2 savings as a result of policies (MtCO2)

201

1 2012 2013 2014 2015 2016 2017 2018 2019 2020

2010-2020

lifetime

Standby 0.5 0.8 1.0 1.2 1.4 1.5 1.6 1.7 1.7 1.8 13.2

ESPUs 0.0 0.0 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.8

Electric Motors

0.1 0.2 0.3 0.4 0.5 0.7 0.8 1.0 1.1 1.3 6.4

Circulators 0.0 0.0 0.0 0.1 0.1 0.2 0.2 0.2 0.3 0.3 1.4

Cold appliances

and labelling

0.1 0.1 0.1 0.2 0.3 0.3 0.4 0.5 0.5 0.6 3.1

TVs and labelling

0.0 0.1 0.2 0.3 0.4 0.4 0.5 0.6 0.6 0.7 3.8

Wet appliances

and labelling

0.0 0.0 0.0 0.0 0.1 0.1 0.2 0.2 0.2 0.2 1.0

Domestic

Lighting 0.1 0.3 0.4 0.5 0.5 0.5 0.5 0.5 0.5 0.5 4.3

Tertiary Lighting

0.4 0.6 0.9 1.0 1.1 1.1 1.2 1.2 1.3 1.3 10.1

Simple STBs

0.2 0.2 0.3 0.3 0.3 0.4 0.4 0.4 0.4 0.4 3.3

5.2 Limitations of the carbon budgets and MTP IA data: the

necessity to explore ‘policy A vs. policy B’ impact

5.2.1 Limitations of the Impact Assessment and carbon budgeting figures

The MTP Impact Assessments attempt to quantify the net benefits of policies by looking at

the cost of implementation to manufacturers (through making products compliant) and

consumers (through increased product costs). This is compared to the benefits the policy

would bring (in terms of the value of additional energy and CO2 savings, and the value of

air quality damage avoided) to obtain an overall net benefit of the policy.

Other government policies which are expected to affect the energy efficiency of products in

each product group are included in the ‗baseline scenario‘ in each Impact Assessment to

ensure that neither the costs nor the benefits are double counted.

Most interview respondents felt that MTP modelling produced reliable and robust data

which they could trust. Interview respondents also mentioned the effect that central

government backing had upon the perceived quality of the data MTP produces.

However, some of those closest to the review and use of the IAs expressed some

reservations:

That the IAs are all ex-ante and there is insufficient ongoing tracking (though this

is being explored). A significant number of the GSBNs which are relevant to the

Impact Assessments appear to have been first published some time ago (some as

far back as 2008), although most appear to have been updated or reviewed in

2010 or 2011.


Final report

April 2012

23

It can be difficult to establish from the IAs what is assertion and what is evidence

based. As with any modelling, there are limitations to the assumptions that might

underpin parts of MTP‘s impact assessments. Assumptions need to be made about

the future of fast-changing markets which, even when based on the best available

evidence, can deviate significantly from reality.

The savings are based on average annual savings over 10 years; therefore where a

small timing change has been enacted, this would make very little difference from

the reference scenario e.g. on standby.

There is an assumption that extra savings are as cost effective as the baseline

would have been; this does not seem a reasonable assumption.

Some research participants, particularly industry representatives, felt that some

modelling assumptions had been made while only considering part of the relevant

product market, rather than all of it. These interviewees did concede, however,

that getting the relevant data from their members in order to provide holistic

market representation can be difficult.

As well as these comments from the consortium interviews, some additional limitations –

specific to the MTP evidence base - were identified through an internal Defra review:

Benefits from minimum energy performance standards (MEPS) seem to increase

past 2020 despite rising energy prices. This has been highlighted as being contrary

to economic principles and suggests the modelling does not take account of the

interaction between energy prices and sales.

Costs to manufacturers upon the introduction of MEPS decline more sharply than

intuition suggests they should.

In some cases, little sensitivity analysis has been conducted to assess what would

happen to the modelled outputs if some assumptions were to change.

The models do not have the flexibility for analysis to be re-run with different input

values quickly.

It is sometimes unclear what data sources have been used or where assumptions

have come from.

Finally, having reviewed the data sources, our additional comments would be:

The policy impact cited in IAs is based upon a reference scenario of no policy being

introduced at all. Such a scenario is very unlikely. It is far more likely that a less

stringent policy would have been introduced, or a policy would have been

introduced later. This is the key driver for considering a ‘policy A vs policy B’

scenario.

The baseline for product groups is usually a straight line extrapolation of past

trends; the justification for this is not clear.

Looking at the impacts for product groups in isolation ignores real-world effects

that might mean increased use of one influences decreased use of another. For

example, heating and air conditioning.

The general principle of an s-shaped curve for impacts between 2010 and 2020

seems intuitive and sensible, because it will take some time for the benefits to be

fully achieved (standards complied with) and these will plateau as the market

becomes fully compliant with a policy.

However, the curve does not account for ‗ramping down‘ of policy impact over time

i.e. one has to assume that the market may have reached the standard regardless

of a specific policy; it may not be realistic to assume that net benefits will continue


Final report

April 2012

24

to increase all the way up to 2020 and will then be consistent beyond that point, as

seems to be the case in most impact assessment curves.

It is important that the reader is aware of these limitations. However, the carbon budget

figures – partly derived from the IAs - have been used as the basis to calculate the

impacts of MTP-influenced policies because:

a. They have taken account of some of the limitations highlighted above.

b. In terms of ascertaining UK impact, the IAs are based upon primary data that is at

least as robust as that of other datasets that either Defra or other stakeholders

were aware of4.

c. The IAs and carbon budget figures have taken into account – and attempted to

address – issues that could have arisen from simply aggregating impact

assessment data e.g. policy overlap and years from which figures are drawn.

d. The IAs and carbon budget figures disaggregate impacts by product group in a way

that fits with the attribution approach.

In addition, Defra representatives felt that the standard of IA - in terms of transparency

and publishing – has improved recently, and that more conservative assumptions are

starting to be used, driven by the need for carbon budgeting.

For comparison, the lifetime impacts from the IAs explored was compared to the predicted

impacts of the E3 standards and labelling Program in Australia and New Zealand. The

estimated lifetime economic benefit of this was estimated at AUD$22.4 billion by 2024 plus

NZD$5.1 billion2 to the New Zealand economy by 2036. The UK impact assessment

estimates therefore seem proportionate and within the right order of magnitude when it is

considered that:

The IA data in the table covers only the product groups that have been selected for

attribution evaluation; the total lifetime figure taking into account all products is

larger.

The E3 impact reflects business and industry benefits as well.

5.2.2 Rationale for establishing a ‘policy A vs policy B’ scenario

In summary, although there are significant limitations to the IA data that underpin the

calculations of MTP impact and value for money in this evaluation, it was generally agreed

that these figures were the best available data and that the IAs were broadly accurate in

assessing MEPS & labelling policy impact against a counterfactual of no policy at all.

However, as MTP have rarely influenced the idea of introducing a policy for a particular

group, it would be a substantial overestimate if the current IA and departmental figures

were used as a guide to the impact directly influenced by MTP.

For example, for standby, MTP data was predominantly used in order to secure the

introduction of more stringent standards 1-2 years earlier than would otherwise have been

4 A secondary aim of the phase 2 interviews was to allow stakeholders the opportunity to cite any

alternative impact figures that might be more credible than those in the MTP IA documents. Although

some stakeholders expressed reservations about the way in which the MTP impact assessment and

forecasting model data may have been calculated, they generally conceded that this data was ‗as good

as it gets‘.


Final report

April 2012

25

the case. A MEPS policy would have been introduced regardless and more stringent

standards would almost certainly have been in place by the third year from the

introduction of the MEPS. It is therefore misleading to suggest that MTP influenced

anything more than the marginal impacts derived from two years of having more stringent

requirements, as the vast majority of the lifetime savings from the policy would have been

delivered anyway. Yet the IA does not take account of this.

Therefore, as well as showing the level of MTP influence against total ‘policy vs.

reference’ impacts, section 6 of this report also attempts to provide a more

accurate estimate of actual MTP impact using ‘policy A vs. policy B’ impacts i.e.

the marginal / additional impact of the MTP-influenced policy outcome compared

to the policy outcome that would otherwise have been realised.

This ‗policy A vs. policy B‘ impact was established through:

a. Using the qualitative data on attribution to understand how the MTP-influenced

policy differed from the policy that would otherwise have been introduced.

b. Using the section of the SEUP benefits document which estimates a portion of the

overall ‗policy vs. reference‘ impact which MTP had a direct influence upon. As

described in section 4, MTPs internal ‗SEUP Benefits‘ document5 estimates the

extent to which the impacts achieved by policy outcomes for each product group

are attributable to Defra / MTP contributions. This effectively points to a policy B /

counterfactual policy outcome that would have occurred without Defra / MTP input.

Estimates were reported as % ranges i.e. Defra / MTP can claim between x% and

y% of the impacts derived from the policy outcome for product group z. The mid-

point of this % range was applied to the carbon budget impact figures for each

product group to provide an estimate of the additional impact of the MTP-

influenced policy compared to the policy that would otherwise have arisen i.e.

‗Policy A vs Policy B‘. Using this approach ensures a more conservative

estimate of total impact.

The % ranges shown in the SEUP Benefits document are as follows:

Implementing Measure Percentage

influence in EU

Standby 25% - 30%

ESPUs 7.5% - 12.5%

Electric Motors 20% - 30%

Circulators 30% – 50%

Cold + Label 1% to 3%

TVs + Label 20% - 30%

Wet + Label 5% - 10%

Domestic Lighting 25% - 35%

Tertiary Lighting 7.5% - 12.5%

Simple STBs 15% - 25%

5 This document was produced by Andy Cotterill and Davide Minotti as an attempt to quantify the

impact MTP has achieved through its work. The impact figures presented in the document are based

on the impact assessments used in MTP‘s modelling work.


Final report

April 2012

26

5.3 Other policy impacts

In addition to input into the setting of standards for the above product areas, phase 1

interviews identified other policies to which MTP may have contributed. The impacts of

these policies are discussed below.

There are several issues regarding the impact data available for these policies:

1. Since the majority of the impact figures discussed below are EU-wide impacts

rather than UK-specific impacts, disaggregation to a UK impact has not been

possible.

2. The basis for some of the calculations is not clear.

3. The ranges on some of the impact estimates indicate that these estimates may not

be robust.

On this basis, MTP influence on most of these policies is discussed without attempting to

quantify their UK impact.

5.3.1 Voluntary agreements

Voluntary agreements take the form of European Codes of Conduct and individual

agreements with particular industry sectors. Voluntary agreements can also take the place

of a regulation under the EU Ecodesign Directive e.g. complex set top boxes and imaging

equipment. A list of current codes of conduct is given below, along with an estimate of EU-

wide savings resulting from the code (or, in the case of the code of conduct for data

centres, UK-wide savings):

Code of Conduct for Data Centres – estimated UK savings from this code are

4.7MtCO2 between 2008 and 2014.6

Code of Conduct for Digital TV Services – an EU wide implementation of this code

could result in an overall reduction in energy consumption of 8TWh per year in the

EU, an equivalent saving of around €750m per year.7

Code of Conduct on Energy Consumption of Broadband Communication Equipment

- an EU wide implementation of this code could result in an overall reduction in

energy consumption of 25TWh per year in the EU, an equivalent saving of around

€7.5bn per year.8

Code of Conduct on Efficiency of External Power Supplies – this code of conduct

will result in a maximum reduction in electricity usage in the EU of 5TWh per year,

equivalent to a total saving of €500m per year.9

Code of Conduct on AC Uninterruptible Power Systems (UPS) – no estimated EU-

wide or UK-wide savings for this code of conduct were available.

6 http://efficient-products.defra.gov.uk/cms/eu-code-of-conduct-for-data-centres-launched/ 7 http://re.jrc.ec.europa.eu/energyefficiency/pdf/CoC_Digital_TV-version%208_2009.pdf 8

http://re.jrc.ec.europa.eu/energyefficiency/pdf/CoC%20Brodband%20Equipment/Code%20of%20Con

duct%20Broadband%20Equipment%20V4%20final%2010.2.2011.pdf 9 http://re.jrc.ec.europa.eu/energyefficiency/pdf/CoC_Power_Supplies_Version4-March2009.pdf

http://efficient-products.defra.gov.uk/cms/eu-code-of-conduct-for-data-centres-launched/

http://re.jrc.ec.europa.eu/energyefficiency/pdf/CoC_Digital_TV-version%208_2009.pdf

http://re.jrc.ec.europa.eu/energyefficiency/pdf/CoC%20Brodband%20Equipment/Code%20of%20Conduct%20Broadband%20Equipment%20V4%20final%2010.2.2011.pdf

http://re.jrc.ec.europa.eu/energyefficiency/pdf/CoC%20Brodband%20Equipment/Code%20of%20Conduct%20Broadband%20Equipment%20V4%20final%2010.2.2011.pdf

http://re.jrc.ec.europa.eu/energyefficiency/pdf/CoC_Power_Supplies_Version4-March2009.pdf


Final report

April 2012

27

5.4 Green Procurement

Government Buying Standards cover the minimum performance of energy using products

which can be purchased by government. Currently only one impact assessment of the

effect of this policy has been undertaken, relating specifically to ICT equipment. This

impact assessment estimates the net benefit of the policy on consumer savings to be

£24m over the next 10 years10.

5.5 Other

MTP has also provided some information and data to help develop the Carbon Emissions

Reduction Target (CERT) and Energy Saving Trust Recommended (ESTR) schemes.

These initiatives have not been explored at the attribution stage as there is limited data

relating to their individual impacts on the UK, and attribution of any impacts to MTP was

considered to be negligible by phase 1 respondents.

5.6 Wider benefits

5.6.1 European Benefits

The internal ‗SEUP Benefits‘ document attempted to quantify the additional impact that EU

policies will have Europe-wide, by calculating a rough ‗approximating factor‘ to scale up UK

savings to a European level. This approximating (extrapolating) factor is derived by

dividing EU carbon emissions by UK emissions. The document acknowledges that this

method of estimating EU-wide savings is only illustrative, and relies on a number of

assumptions, some of which are clearly not without issues; these assumptions include:

That the relationship between energy and CO2 impact is constant throughout the

EU;

That there is similar potential for energy efficiency improvements in products

throughout the EU;

That sales and use of products is constant throughout the EU;

That there are relative uniform prices (e.g. for energy, values of CO2) across the

EU.

In addition, applying the same factor to all product areas implies that EU member states

will be impacted to the same degree as the UK by the policies in each product group. In

reality there are likely to be product areas which impact some member states more than

others depending on the relative abundance of each product within that country. Despite

the limitations of this method and inherent complexity of producing EU-wide impacts for all

product groups, it is clear that EU energy efficiency standards will have some impact on

other EU member states in the same way in which impacts have been quantified for the

UK.

10 Based upon figures provided by Defra in the document ―Impact Assessment of Revised

‗Government Buying Standards‘ specifications for Information and Communications Technology –

Computers‖. It is important to note that although this document is still a draft, the impact figures in it

are not expected to change. The £24m is a midpoint between the predicted impact range of £16m and

£32m.


Final report

April 2012

28

5.6.2 Global impacts

The Energy Star programme also influences benefits beyond the UK. For example, Energy

Star estimates that in the last five years, the US Energy Star programme for office

equipment resulted in savings of more than 223TWh and energy bill savings of $22 billion

for the US. It also estimates that in the last three years, the EU Energy Star office

equipment programme is estimated to have resulted in savings of around 10TWh, or more

than 2 billion euros11. MTP provides independent data to the programme to support the

development of product standards criteria.

MTP has contributed to the International Energy Agency‘s 4E‘s programme (Implementing

Agreement for a Co-operating Programme on Efficient Electrical End-Use Equipment). The

programme‘s Mapping and Benchmarking annex makes use of MTP data and publications

in order to achieve its aims:

The mapping provides an overview of the energy efficiency performance of certain

electrical end-use equipment (or energy using products) in several countries

and/or regional areas and a brief summary of the main policy measures regarding

energy efficiency of products in those areas;

The benchmarking compares average and best performance of products put on the

market in different countries and/or regional areas; analysing variations between

different markets; sharing best practices and lessons learnt by, for example,

highlighting potential for further improvement of energy efficiency levels globally.12

MTP also provides data and evidence to support the efforts of the Super-Efficient

Equipment and Appliance Deployment (SEAD) in influencing transformation of global

markets.

5.6.3 MTP work achieving non-carbon impacts

The work of MTP in influencing UK and EU product policy has other potential impacts aside

from the reduction in carbon emissions and UK consumer savings associated with an

increase in efficient products in the market. These include raw material savings, water

savings and additional revenue and market share increases associated with the

introduction of these products.

As well as these more tangible benefits, some interviewees felt that by using MTP‘s

outputs, Defra was improving the UK‘s international reputation and / or influencing other

countries to carry out similar modelling on their own product markets, leading to better

standards of efficiency beyond the UK.

Although these non-carbon impacts should be considered as part of the overall value of the

MTP, they have not been quantified or included in the value for money analysis.

11 Promotional email from the Energy Star programme 12 http://mappingandbenchmarking.iea-4e.org/

http://mappingandbenchmarking.iea-4e.org/


Final report

April 2012

29

6 Attribution: Evidence of MTP influence

6.1 Stakeholder awareness of and engagement with MTP.

As a criteria of their selection, all phase 2 stakeholder respondents were aware of MTP

(though respondents did note that some of their colleagues / other member states may

not be). However, the majority of respondents at the Commission and from other Member

States had limited direct contact with MTP, reflecting the division of responsibilities under

the current contract.

Industry representatives‘ main contact with MTP arises through consultations issued by

Defra. A few respondents also noted further engagement around data sharing but this still

tends to be largely within the context of a consultation response. All industry respondents

have an on-going dialogue with Defra involving regular meetings with Defra

representatives.

Respondents‘ understanding of the distinction between MTP and Defra was mixed. Many

respondents at the Commission were unclear if MTP was a separate programme or whether

it was a section of Defra itself. Often respondents were not clear if they were engaging

with someone from Defra or MTP, though this does not seem to have influenced their

perception of the effectiveness of Defra or MTP. Industry respondents in UK trade bodies

were clearer on the distinction.

6.1.1 Perceptions of MTP

Respondents‘ perceptions of MTP were generally positive, though given the limited direct

contact of many with MTP, few respondents were able to comment authoritatively. Some

commented more generally about the objectives of a programme such as MTP rather than

MTP‘s work in particular.

Member State and Commission representatives were generally very positive. In particular,

MTP was praised for its independence, technical capability and the extent of its evidence

base and research. The long term strategic approach to developing the evidence base was

also commended. The individuals working on MTP were seen as at least competent and

often expert in their field. Two respondents felt that the MTP research and evidence base is

more extensive than that developed by equivalent sources for other member states.

Positive statements from European-level respondents regarding MTP included the

following:

“The UK, through MTP, has the most extensive database. It is a serious partner,

also trusted by industry. MTP has a good line between industry and member

states. They provide good technical input.”

“MTP is valuable as it is the most complete source; it‟s seen as a reliable and

government-backed official source that can be trusted.”

―An excellent evidence base. No other country undertakes this type of research.”

―It is a thoughtful and well set up programme. It is good that it had a very

consistent philosophy on the role of evidence making.‖


Final report

April 2012

30

―A good structure and long term plan which is very useful for focusing on technical

information and data which is very important and lacking at the EU level.....there is

a clear advantage of MTP, who have a continuous database that they are building

up. Here we could learn from MTP.”

The industry view of MTP was more mixed. MTP was praised for its engagement with

industry and noted for its independence. However, one industry respondent was negative

about MTP‘s work, feeling that their association was having to drive the moves towards

adoption of higher standards in the sector, with little or no enthusiasm or technical input

from MTP (who they felt were under-resourced). Positive responses from industry

representatives included:

“MTP play a very useful role. They are very responsive and very active. They

engage well with industry.‖

“It is good to have a programme to show actors in the market that there are

alternatives with a better environmental footprint and that they usually make

economic sense as well. There is a need to cover the „death valley‟ between proof

of concept and market adoption.‖

“Broadly, the MTP process as a whole is healthy. I think the fact that Defra

established an independent evidence base is good. I think the fact that it is

consultancy-based is reasonable – I can‟t judge if outsourced consultancy is better

than in house. The people we deal with are genuine and competent.‖

6.2 Verification of MTP influence on MEPS and labelling

6.2.1 Phase 1 anticipated influence

Minimum energy performance standards were identified as likely to be the area where MTP

has had the clearest positive impact, through Defra and the Commission using the MTP

evidence base to:

1. Prompt the development of new standards for product groups which otherwise

would not have minimum standards set;

2. Influence higher standards to be set by the Commission, usually through increased

scope or more stringent requirements;

3. Bring forward the timescale for introduction of standards that would have been

introduced anyway but less quickly;

4. Influence the maintenance of a standard proposed by the Commission, which

would otherwise have been lowered/diluted due to influencing agendas of other

member states.

Labelling follows the same regulatory process as MEPs and has been evaluated alongside

them, but where possible impacts have been identified separately as they act on product

markets in a different way i.e. more on demand side. MTP is likely to be having an impact

on labelling by:

1. Influencing the introduction of labelling on products that would otherwise not have

had labelling;

2. Influencing the energy performances pertaining to different labelling

categorisations;


Final report

April 2012

31

3. Influencing the format and structure of the label to make them clearer for

consumers;

4. Influencing to maintain energy performance labelling categorisation standards

proposed by the Commission, which would otherwise have been lowered/diluted.

6.2.2 Acknowledged general UK / MTP influence

The UK was noted by almost all member states and Commission representatives as being

amongst the member states most consistently calling for more stringent requirements or

for earlier adoption of measures and is generally viewed as making a contribution to the

outcome of negotiations in most areas to some degree.

UK representatives to the Commission are highly regarded by all respondents. They were

described as active, authoritative, constructive, well informed and good at engaging

stakeholders. The UK was repeatedly identified as one of a small number of very active

and vocal Member States; other cited were the Netherlands, Germany, Italy, Sweden, and

Denmark:

[The UK is] one of, if not the most active of, Member States.‖

“Together with the Netherlands and sometimes Sweden, the UK were the most

constructive Member State. They have a really in depth knowledge of the subject.

Their comments were extremely well founded, and constructive, they have solid

data to back up their arguments. This proved to be extremely useful for the

Commission. In the end it improved the quality of the regulation.”

―The UK delegation is well informed, gives clear opinions and proposals to push

things forwards.”

―There is no doubt that the UK is one of the active Member States on both Eco-

design measures and Energy Labelling (and Energy Star too) and in most cases

advocates more stringent levels or a more early adoption. Although such action is

certainly needed, it does not always result in changes because other Member

States are advocating in the other direction.‖

―The UK government representatives are well respected in the EU and in

Commission work. There is an advantage for them in terms of both language and

in a long experience of stakeholder consultation. Whenever Defra speak it is

authoritative. The feedback I get is that there are only four to five member states

speaking up and Defra is one of them. Defra acts as a counter weight against large

member states that have a different agenda.‖

One Commission respondent commented that the UK almost always inputs and even if it

does not get a majority vote will likely be able to secure a blocking minority. Respondents

felt that this position partly reflected the fact that the UK does not a have a large

manufacturing base and that other Member States were more protective of national

competitiveness reflecting large national manufacturing interests. Another noted that it

reflected that the responsibility for product policy sat within the UK environment ministry

and therefore was more influenced by ambitions under climate change policy.

Furthermore, the evidence base possessed by Defra was viewed by the Commission,

member states and Defra themselves as being a contributory factor to the strength of the

UK contribution. Those involved in negotiations felt that MTP data has played an important

role in supporting Defra to defend their own position in the face of opposition from other


Final report

April 2012

32

Member States, though it is not clear the extent to which this is also supporting a common

position with the Commission. The UK was cited by several member state respondents as

backing up their position with data and using that to be constructive; this was not felt to

be the case for all Member States. The UK was noted for being specific about the

requirements they were seeking; again due to the work conducted. Respondents felt that

the evidence base was an advantage for Defra and strengthened their position.

“[The UK is] trying to use evidence as the basis for their positions and I think in

this the MTP was quite useful to sharpen their arguments.”

―It is already quite an advantage that the UK gathers this evidence. Often the

positions of some members states are purely based on the input they get from

industry; not that the UK does not listen to industry. But there is this added value

here of trying to make this evidence based policy making.‖

―It adds a lot of weight to the debate – it is a transparent approach, which is open

for all to see.”

―It [the UK input] was very technical in a positive sense, very focused on product

and issues that could come up, definitions. Very informed, very detailed technical

knowledge and analysis. The UK is making a strong contribution with strong

technical content.”

―[The UK is] quite influential; on one hand because of being large and having lots

of votes, but normally they give constructive comments based on data, which you

don‟t have in every case. It is to do with their ability to provide numbers, to

specify the precise value they want for requirements, not just call for it to be

stricter in general.”

One respondent noted that the UK used the evidence base in a way that it was able to

change the nature of the debate. The UK representatives present the evidence and then

invite discussion around the findings. Therefore it is not simply a debate around the

validity of the UK position but also one about the evidence base itself: ―They did not have

to bring forth arguments like others, the normal diplomatic wheeling and dealing. Their

evidence base elevated the discussion beyond a power struggle with the votes.‖

Another noted the importance of the technical evidence base in adding to confidence in the

UK position and authority at the negotiating table; the UK was more likely to be heard in

the debate even if other states might theoretically disagree with the UK. Other

respondents noted more generally the importance of an evidence base for being

influential: ―Having evidence is very important. You normally can‟t change Commission

proposals without the evidence that it is possible to have stricter requirements without

compromising function or products or the structure of the market.”

In several areas MTP data has supported the Commission by filling a gap in the evidence

base or providing more up to date data: ―When there are new products, products that

have not been dealt with before, there are very large gaps in the technical data available

and often we are scrabbling around for information. Commission preparatory studies are

often a collection of what information is already out there rather than commissioning new

data collection and scientific studies.‖

Several respondents noted the lack of available data when putting together preparatory

studies and that the Commission is often reliant on data from member states or industry

when it comes to negotiations that might take place several years later. Regarding the


Final report

April 2012

33

latter source, respondents commented on the value of having a view independent of

industry, who would be aware how data was likely to be used: ―Having the technical input

was very important, for the Commission too. The regulations are based on the preparatory

studies and these are sometimes finished two or three years before the adoption

procedure so the data can often be out of date. We are counting on data provided by

industry and they always downgrade the real efficiency improvement potential as they

know the information they give will be used for setting legal requirements. So it is useful

for the Commission to have an outside independent evaluation which is up to date at the

time of negotiation. And either confirms or pushes upwards the requirements we adopt.‖

One respondent identified an important issue that the current Commission requirement –

being pushed by the UK - for centralised EU-wide data to be made available is not being

met and there is no clear responsibility for who should coordinate it. The Directive states it

should be provided by manufacturers on free and accessible websites but someone needs

to collect it. The Commission is planning to centralise collection of product test data with

the development of a database where manufacturers will be encouraged to voluntarily

submit their product data for new products. The aim is for the market to push action by

manufacturers, though the degree to which this is achieved and the speed at which a

substantial volume of accurate data can be collated is not clear.

6.2.3 Barriers to identifying and achieving influence

Despite this general affirmation of the scale and authority of UK input (and the importance

of the MTP data in enabling this), most respondents from Member States were unwilling or

unable to give a clear statement that the UK had been responsible for specific outcomes.

All commented on the consensus-based nature of the negotiations and the balance of

technical and political arguments.

The process takes on average four years to complete; two years for the preparatory study

and two years for consultation and negotiation. In particular, the consistently divergent

views of Member States with or without manufacturing interests was noted as a common

theme of most debates and this always necessitates the need to move to a compromise

position that was acceptable to a majority of parties. The challenge of attributing specific

policy outcomes to the UK, let alone MTP, was articulated by respondents as follows:

“It is difficult - or even impossible - to verify claimed impacts of one specific actor

in these processes...in general any impact is „group work‟.‖

“It is collaboration. Even if you disagree, as you discuss you find a compromise

which is a good solution for both countries. All the information the UK provides is

considered but it is not possible to say if any UK information is considered more

than others.‖

Another reason respondents at the Commission appeared to be unwilling to comment too

explicitly on the UK influence was the politically sensitive nature of commenting on the

influence of any one Member State. One explicitly stated he was unable to comment on the

basis of confidentiality linked with the commitology procedure; others made similar

comments.


Final report

April 2012

34

A few respondents from the Commission and Member States were more direct about the

influence of the UK but were still unable to be very specific in identifying examples of the

UK influencing a particular outcome.

“The UK provide clear positions and lots of ideas. They are one of the few active

Member States. They are perceived as one of the driving forces in these

processes.”

“A position is always shaped by the UK.”

As well as highlighting unwillingness to directly attribute policy outcomes to the UK, the

interviews identified limitations to the extent to which Defra can use MTP‘s work to

influence EU policy are limited by a number of factors:

Some EU member states – especially those with large manufacturing bases –

approach the EU negotiations with different priorities to the UK. Possession of a

strong evidence base may be insufficient in the face of national priorities.

Inevitably, much of MTP‘s evidence base work is UK-focused and designed to

ascertain the benefits of particular policies to the UK market, therefore the extent

to which it is felt to be authoritative outside the UK may be affected by differences

in product use, predominant product models and the extent of other demographic

differences.

Where there have been concerns about MTP modelling or assumptions, the

evidence base was less likely to be used or pushed forward in negotiations,

therefore less likely to influence.

There are also likely to be instances where the MTP data has endorsed what the Defra

position would have been anyway, or where the Defra position has aligned with the most

likely outcome at European Commission level. In these instances, evidence from MTP will

only have been used to endorse an outcome that probably would have occurred anyway,

and so would not result in any additional impacts.

6.2.4 Exploring the counterfactual

To attempt to overcome respondents‘ inability to precisely assess the influence of the UK –

and MTP – on policy outcomes, respondents were asked about their views on how

effective Defra (or the UK) would have been if operating without the evidence base.

Respondents commented that the UK would not be as effective without the technical

evidence base they currently have. Respondents felt that without MTP, the UK would have

to default to a political negotiating position and rely more on gathering support from

industry: ―I think they would have negotiated like other big member states, to gather

people to support their case and not necessarily back things up. They did not have to bring

forth the normal diplomatic wheeling and dealing.” Another respondent went further,

stating that the lack of an evidence base could mean the member state cannot exercise

much influence: “Those that can afford it invest in technical resources. This is very

important if a Member State is to be active in the discussions.‖

It was noted by some respondents that the UK has a large number of votes and that other

Member States without MTP-type technical support can also be active. However, given the

counter discussion on the importance of the evidence base for the UK position it can be

inferred that the UK would not be in such a strong position as States that have no evidence


Final report

April 2012

35

base but have a national interest to defend and are pushing for maintaining the status

quo. It was certainly felt that the UK would not be able to specify a specific position that

benefitted a progressive policy agenda.

It was also felt that the Commission would lose an important and consistent data source.

Although one respondent felt that the UK itself would not feel the impacts of a world

without MTP for around two years, they did state that the Commission would feel the loss

sooner.

Another source of insight into the importance of MTP is provided by comparison with the

situation in other Member States. The UK is unique amongst the other Member States

interviewed in having a technical resource that is structured and funded like the Market

Transformation Programme. However, in many cases, Member State representatives will

also have technical support in the meetings with the Commission. Where there is a

national energy agency, the technical experts are from that agency. If there is no agency,

the ministry representative tends to bring external experts. Where their support is limited

in terms of resource, there may be more collaboration with like-minded Member States.

Germany, Italy and the Netherlands all have technical support provided by a government

agency, though it was also noted by some respondents that Italy‘s manufacturing

companies may prepare evidence bases to demonstrate the implementation costs / issues

around certain standards.

Therefore, in comparison, some Member States who are also proactive in the discussions

at the Commission have also invested in a technical resource, but this is usually within the

existing government administration / energy related agencies. The scale of that investment

tends to reflect the scale of the national manufacturing base for energy using products.

This also influences which part of the government lead on the discussions, with economics

ministries leading where there is a large manufacturing base and environment ministries

leading where it is predominantly a consuming country. Where there is less investment

and a limited technical resource, this is strengthened by building closer links with similar

minded Member States.

6.2.5 Specific influences at European level

As well as strengthening the data input to the regulatory process, MTP has also

strengthened the technical capability. One Commission respondent noted that the technical

input from MTP for standby and TVs has been influential in clarifying queries and speeding

up the regulatory process.

For ICT products - computers, displays and imaging equipment - MTP is currently providing

additional technical staff to work directly for the Commission given the limited resources in

the Commission and a delay after the preparatory study. MTP felt that this was a good way

of influencing the Commission and being able to push for more stringent requirements and

was able to help the Commission distinguish technical from political views. Overall, it is not

clear to what extent MTP has contributed to more stringent outcomes in this area as the

process is not yet finished. However, it is clear that MTP has made a positive contribution

enabling the Commission to develop their proposal in the direction of more stringent

proposals.


Final report

April 2012

36

MTP is noted by several respondents for its input to the development of test standards for

verification and compliance testing. This was noted for standby with Bob Harrison (an

independent MTP expert consultant in the area of consumer electronics) being noted

particularly for his contribution to test standards for simple set top boxes and standby. The

contribution to test standards for TVs was also noted but there was some uncertainty

around the specific outcomes claimed by MTP. Other specific contributions to verification

and test standards were verified - to varying degrees - for wet appliances. MTP also

claimed a contribution to test standards on EPSUs, electric motors and tertiary lighting

although this was not verified by any other respondents.

Many Commission respondents noted that Defra has been very active on market

surveillance although Defra funds the NMO to principally overseeing compliance at the UK

level.

6.2.6 Refutation of influence and critiques of MTP data and the UK position

One Member State respondent felt that the UK position had at times focused too strongly

on technical points that, in their view and from their own analysis, were of marginal or

even negligible impact in terms of energy savings. They felt that the UK has argued too

strongly and for too long on some of these points and that the technical input from MTP

was partly responsible for this.

6.2.7 Summary of influence on MEPS and labelling

The table below summarises the extent to which MTP impacts claimed in phase 1 regarding

MEPS and labelling have been verified through the phase 2 research:

The extent to which Defra/MTP had influenced the outcome of the policy making process -

both generally and for specific claimed impacts - is rated as follows:

Strong influence - MTP (alone or through Defra) were critical in influencing the

outcome;

Some influence – MTP (alone or through Defra) made some contribution to the

outcome ;

Weak influence – MTP (alone or through Defra) made a slight difference to the

outcome;

No influence - MTP (alone or through Defra) made no difference to the outcome.

The level of confidence in that assessment of influence; established through a ‗traffic light‘

system:

Green – there is good verification of the Defra / MTP claim, with at least one

respondent is able to authoritatively verify the claim and there is a good match in

the detail for specific claims.

Amber – there is some verification of the claim; at least one respondent is able to

verify the claim but there is some uncertainty in the recall of the detail of specific

claims.

Red – there is no verification of the claim made by MTP / Defra or there is

disagreement between respondents or such a large degree of uncertainty that

verification of the actual influence on the outcome is difficult to discern.


Final report

April 2012

37

Hypotheses

Product group Introduced More stringent/

earlier/ speedier Held EC line

Compliance – do

we need this here?

‘Policy vs. Reference’

Impact assigned to

the policy from latest

departmental figures

(average annual

impact)

‘Policy A vs. policy B’

impact assigned from

review of attribution

information and SEUP

% apportionments

(average annual

impact)13

Standby No influence claimed Some influence Some influence Some influence £288m / 1.80MtCO2 £28.8m / 0.18 MtCO2

EPSUs No influence claimed Weak influence No influence claimed Weak influence £11m / 0.1 MtCO2 £1.1m / 0.01 MtCO2

Electric motors No influence claimed Some influence Some influence Some influence £92m / 1.3 MtCO2 £23m / 0.325 MtCO2

Circulators No influence claimed Some influence Some influence No influence claimed £-1114m / 0.3 MtCO2 £-4.4m / 0.12MtCO2

Cold appliances

and labelling No influence claimed Some influence No influence claimed No influence claimed £44m / 0.6 MtCO2 £0.88m / 0.012MtCO2

TVs and

labelling Some influence Strong influence Some influence Strong influence £64m / 0.7 MtCO2 £16m / 0.175MtCO2

Wet appliances

and labelling Some influence Some influence No influence claimed No influence claimed £14m / 0.2 MtCO2 £1.05m / 0.015MtCO2

Domestic

lighting Some influence Some influence No influence claimed No influence claimed £124m / 0.5 MtCO2 £37.2m / 0.15MtCO2

Tertiary lighting Weak influence Weak influence Weak influence No influence claimed £214m / 1.3 MtCO2 £21.4m / 0.13MtCO2

Simple STBs Strong influence Some influence No influence claimed Strong influence £82m / 0.4 MtCO2 £16.4m / 0.08MtCO2

13 Figures for this column show average annual impact except for standby, where figures are fixed because the impact was timing rather than stringency. 14 Despite having a positive environmental impact, EU policies relating to circulators were calculated to have a slight negative financial impact in the UK.


Final report

April 2012

38

The table shows the following:

That where MTP influence has been identified and verified, this influence has

usually been contributory – ‗some influence‘ - rather than critical. No phase 2

respondents were able to identify a critical role for MTP. This partly reflects the

underpinning role of the programme and the complex nature of the regulatory

process it is trying to influence i.e. a balance of political and technical arguments,

multiple stakeholders with varying degrees of involvement and a lengthy

timetable.

That where MTP influence is most often recognised within the consortium, this

tends to be either influencing the stringency of a policy or accelerating its

introduction rather than influencing the establishment of a policy where there

would have been none. Discussion with Defra and MTP in Phase 1 indicated that

there would be less activity in this area since the EC are fairly rigorous in their

listing of products for consideration.

It is also difficult to assess the extent to which MTP has also played a role in

supporting Defra and / or the Commission in holding a position in the face of

opposition from other Member States. This was not a situation generally

recognised by respondents, given the perception of the UK as proactive and

dominant in calling for more stringent or earlier requirements.

The clearest verification of claims from Defra/MTP for this influence was

established for standby, TVs, simple STBs and wet appliances.

6.2.8 Influences on non-priority products

The phase 2 interviews brought to light a number of examples of MTP influence upon

policies affecting products not included in the priority products focused upon in this

evaluation. These comprised the following:

As part of MTP‘s direct support to the Commission on the development of

standards for ICT products, gaming consoles were incorporated into the proposed

regulation. The original Commission proposal was to base the requirements on

Energy Star which only covers office equipment and would have excluded the sale

of higher energy consuming gaming consoles and gaming PCs and possibly led to

illegal imports or imports of components with no regulation on energy efficiency.

For network standby, the Commission desk officer noted the UK has provided a

strong contribution with a strong technical content but mainly around definition

and packaging rather than the level of ambition.

For vacuum cleaners, the UK has been particularly active in the discussions around

measurement of energy consumption. This was noted both by respondents from

the Commission and from industry. The Commission stated the UK input was very

useful and they contributed well.

For air conditioning, a respondent from Germany noted the UK led calls for stricter

requirements with support from other Member States, but was not sure if MTP data

or evidence was used.

6.3 Influence on Public (Green) Procurement

MTP was acknowledged to have played a significant role in the establishment of public

procurement standards in relation to ICT (as an MTP specialist on this product group was

consulted by Defra. For other energy using products, MTP provided some input, though


Final report

April 2012

39

this was less holistic and mainly covered energy use rather than other considerations such

as recyclability. A Defra representative did acknowledge that Defra had not asked MTP for

input beyond energy use.

A Defra respondent reported that initially, the provision of information for public

procurement was an opportunistic move to use information already developed by MTP as

part of their work on MEPS and labelling. Therefore it originally only covered products

under the Ecodesign and Energy Labelling Directives. Provision of this information for

procurement was an ‗add on‘ to the main body of MTP‘s work.

Going forward, an opportunity was identified by Defra to develop a more effective and

structured green public procurement programme. The expertise and access to support

offered by MTP was noted as enabling Defra to set up a proper programme of support.

The respondent currently responsible for this area reported that MTP still provide support

to Defra for energy using products but that these have become smaller aspects of the

procurement standards. MTP now provide cost benefit analyses as required under a

framework agreement. MTP recommendations are generally accepted. These support Defra

in continuing to raise the energy performance requirements in procurement standards

across departments. The respondent felt that MTP are well respected and their input is

generally seen to be accurate and of a high standard.

Overall, MTP has been critical in enabling the introduction of new procurement standards

for ICT. MTP has a continuing role in contributing to more stringent standards on this and

other product groups. This reflects the extent of the evidence base developed for their

work on MEPS and labelling.

Policy Introduced More stringent/ earlier/

speedier

Public green procurement Strong influence Strong influence

6.4 Influence on voluntary agreements

For EU Codes of Conduct, the UK is again noted by Commission and Member State

respondents as one of a few active Member States contributing to the discussion at the

Commission level; Denmark, the Netherlands, Austria and Germany were also cited in this

regard. This has largely been through calling for stricter requirements and test standards

rather than through calling for new product areas to be covered.

The technical consultant to the Commission acknowledged the involvement of MTP in

external power supply units (EPSUs), simple set top boxes (STBs), and broadband

equipment. The relevant desk officer at the Commission recalled UK input to discussions

around upcoming agreements on complex set top boxes and imaging equipment.

Like the discussions with MEPS and labelling, respondents again found it difficult to identify

a specific influence of the UK on the outcome of negotiations in relation to voluntary

agreements. Again this reflected the large number of stakeholders and consensus driven


Final report

April 2012

40

nature of the negotiations. However, the input from MTP was viewed positively, in

particular for its ability to provide an independent view: ―There is good consultative

dialogue amongst many parties, so it is difficult to say who has the share of the outcome.

It [MTP] is a serious partner, also trusted by industry. MTP has a good line to industry.‖

The Commission consultant in this area noted the UK contribution has often been

predominantly through representation by MTP; Defra participation declined from earlier

days although the UK participation overall is less than it was.

MTP is currently providing additional technical staff to work directly for the Commission on

ICT products, given the limited resources in the Commission (also working on MEPS and

Energy Star). Support is being provided on a voluntary agreement for imaging products

and MTP are pursuing for more stringent requirements although the Commission has not

adopted these. It is likely these savings will be further reduced given the arrangements

for managing the agreement. MTP are also writing a mandatory measure in case the

voluntary agreement fails.

The Codes of Conduct have preceded MEPS for several product areas. Influencing

voluntary agreements is also a way in which MTP has then gone on to influence the setting

of MEPS, though this has not always been visible to other stakeholders.

Policy More stringent/ earlier/

speedier

Compliance with the

agreement

Voluntary agreements (EU

codes of conduct) Some influence Some influence

6.5 Influence on Energy Star

There is good evidence that MTP has contributed to more stringent requirements for

Energy Star by providing an additional technical resource directly to the Commission,

enabling them to put together a robust and ambitious proposal.

MTP has provided input to the criteria for the voluntary ‗best in class‘ Energy Star label in

the past through its presence on working groups reviewing proposals from the US

Environmental Protection Agency (EPA) who own the Energy Star. An MTP representative is

currently working directly for the Commission to support them on the revision of Energy

Star standards for computers, displays and imaging equipment (also supporting the

Commission on MEPS and labelling and voluntary agreements for ICT products). MTP is

currently advising the Commission on a more ambitious proposal for displays than that

proposed by the US EPA. MTP claims that it has achieved some of its proposals for

displays (e.g. around sleep and off modes) but the process is not yet finished.

The technical support provided by MTP is welcomed by the Commission. According to the

desk officer at the Commission, the USA provides an estimated 80% of the technical input

to Energy Star, therefore it is considered important that Europe / the Commission is able

to provide technical input as well. The Commission Desk Officer reported that the input is

„taken on board and is useful‟ but does not personally get involved with the technical

detail. He commented that the UK is one of four to five Member States that provide input


Final report

April 2012

41

but this is not a controversial area as there is not such a large European Manufacturing

base. The debate is more about what is achievable and cost effective from a technical point

of view.

The technical consultant to the Commission was more positive and felt that the input from

MTP has influenced at least part of the increased stringency of the standard: ―Most of the

influence will be by having good technical arguments and analysis that supports the

arguments. It is the product of many people discussing so it‟s difficult. But I believe they

[MTP] have supported the process and part of the outcome is because of MTP‟s input.

Compared with input from other member states I would say MTP has generally provided

the most useful analysis of the issues. Not just providing opinions but providing analysis in

spreadsheets.‖

Overall, MTP has contributed to more stringent requirements for Energy Star, which

started out as very much a standard with minimum requirements. This has been

particularly strong where MTP is working directly for the Commission.

Energy Star also has a link to the Minimum Energy Performance Standards in that it has

formed the basis of some MEPS for ICT.

Policy More stringent/ earlier/ speedier

Energy Star Some influence

6.6 Other UK-specific policy initiatives

Further policy initiatives, in particular UK policies, were discussed in phase 1 but have not

been explored specifically in phase 2 as they were considered out of scope or of minor

importance / impact. MTP and respondents in phase 2 did identify a limited number of

impacts on UK policy initiatives and these are presented here.

For motors, MTP claimed an influence on the Carbon Trust‘s updating of the Enhanced

Capital Allowance (ECA) scheme with the new IEC testing standards, the same as those

used for Ecodesign and Energy Labelling measures. This effectively makes the ECA more

ambitious than previously and the measurement methods used are more accurate. There

was no verification of this claim.

For circulators, MTP claim their work on circulators led to industry proposing to the

Department for Communities and Local Government (CLG) that they include in their 2010

building regulations a requirement that circulators are labelled with the industry developed

A-G label. This was included in the regulations published in March and is expected to

promote the sales of more efficient circulators until 2013, when the Ecodesign minimum

standard will require circulators to be at least A rated. The monitoring and reporting of

market share of products in this group was acknowledged.

For simple set top boxes, MTP contributed to the work on identification of best practice

standards through collaboration with EST and Digital Tick. This initiative gave recognition


Final report

April 2012

42

to the best performing products in the market (top 10-15%). The standards were

developed in discussion with MTP consultants and criteria peer reviewed by Bob Harrison

and others. This was a demand side measure designed to help consumers choose the most

efficient products on the market. There was no verification of this claim.

The respondent from the Energy Saving Trust stated that MTP data is used for general

savings calculations for all household appliances, research, the Energy Saving Trust

Recommended scheme and the UK Voluntary Retailer Initiative on the TVs. EST use the

briefing notes, in particular the reference scenario and base case, and current market

average performance usage data. The data is useful partly because it is the main and most

complete source: “There is not much else out there.”

The respondent stated that part of the benefit of using MTP data is that it saves EST

calculating savings from first principles based on information from the energy label. If MTP

was not there, EST would probably have to commission testing of more products and

commission more research (for which they do not have funds). MTP provides a reliable and

government-backed official source that they felt can be trusted.

As stated, MTP data is used directly for Energy Saving Trust Recommended label to

provide the baseline saving against which the proposed ESTR standard is compared. MTP

also receive test data from ESTR.

MTP were also involved in the TV Voluntary Retailer Initiative. However, the main contact

for this at EST has since left and could not be contacted. The respondent at EST thought

MTP supported the project and may have provided some of the calculations.

MTP data was also a key input to their recent publication „The Elephant in the Living

Room‟; this was produced partly to access the wealth of evidence held by MTP and make it

more readable and accessible to a wider audience. This is in a similar vein to two earlier

publications - „Rise of the Machines‟ and ‗the ampere strikes back‟. These were joint

initiatives between MTP and EST and were developed in part to showcase the public data

that MTP held.


Final report

April 2012

43

6.7 Assigning impacts to attribution bands

The table below takes all the impacts associated with the various policy initiatives and

assigns them to attribution bands. For each assignment, the traffic light system also shows

the level of confidence in this assignment, based upon the factors set out in the attribution

methodology section.

The figures in the table refer to the overall saving to the UK on each product arising from

introduction of Ecodesign regulations. They do not amount to the unique contribution of

MTP / Defra to the individual products i.e. over and above the contribution of other factors.

The reasons for this are explained in the methodology appendix.

Level of MTP

influence

Product group /

policy

‘Policy vs. reference’

impact (average annual

UK net benefit / net

MtCO2 savings in 2020)

‘Policy A vs. Policy B’

impact (average annual

UK net benefit / net

MtCO2 savings in 2020)

Strong

influence

upon outcome

Green Procurement £2.4m £2.4m

TVs and labelling £64m / 0.7 MtCO2 £16m / 0.175MtCO2

Simple STBs £82m / 0.4 MtCO2 £16.4m / 0.08MtCO2

TOTAL £148.4m / 1.1 MtCO2 £34.8m / 0.255MtCO2

Some

influence

upon outcome

Standby £288m / 1.80MtCO2 £28.8m / 0.18 MtCO2

Energy Star

Voluntary Agreements

Domestic Lighting £124m / 0.5 MtCO2 £37.2m / 0.15MtCO2

Cold appliances and

labelling £44m / 0.6 MtCO2 £0.88m / 0.012MtCO2

Wet appliances and

labelling £14m / 0.2 MtCO2 £1.05m / 0.015MtCO2

Electric Motors £92m / 1.3 MtCO2 £23m / 0.325 MtCO2

Circulators £-11m / 0.3 MtCO2 £-4.4m / 0.12MtCO2

TOTAL

£551m / 4.7 MtCO2, impact

of Energy Star and Voluntary

agreements

£86.53m / 0.802MtCO2

Weak

influence

upon outcome

ESPUs £11m / 0.1 MtCO2 £1.1m / 0.01 MtCO2

Tertiary Lighting £214m / 1.3 MtCO2 £21.4m / 0.13MtCO2

TOTAL £225m / 1.4 MtCO2 £22.5m / 0.14MtCO2

Key:

Green – High confidence in influence assessment

Amber – Medium confidence in influence assessment

Red – Low confidence in influence assessment


Final report

April 2012

44

The charts below summarise the separation of annualised attributed impacts using both

the ‗policy vs. reference‘ and ‗policy A vs. policy B‘ impact figures:

Figure 1: Breakdown of ‘policy vs. reference’ impacts against attribution

categories established in phase 2

Figure 2: Breakdown of ‘policy A vs. policy B’ impacts against attribution

categories established in phase 2

From figure 2, overall, 88% of the CO2 impact influences and 85% of the £ savings

influences which MTP claimed in phase 1 have been verified i.e. the evaluation finds that

MTP had strong or some influence upon the policy outcome.


Final report

April 2012

45

6.8 Value for money analysis

This section uses the data collated in sections 5 and 6 of this report to provide an

assessment of the value for money being delivered by the MTP.

6.8.1 Factors considered in the analysis

The costs and benefits that have been taken into account to enable the calculation are as

follows:

Value for money is calculated using both the ‗policy vs. reference‘ and ‗policy A vs.

policy B‘ figures.

The lifetime – rather than annual - benefits of the policies are used. The lifetime

factors work in the same way for both ‗policy vs. reference‘ and ‗policy A vs. policy

B‘ impacts, with the exception of standby. For standby, as the marginal additional

impact relates to quicker implementation rather than increased stringency, only

the additional impact - derived from the two years where the more stringent

standards might not have been in place without MTP – are used.

Lifetime £ benefits are derived from the annual attributed impacts table in section

6.7 and are assigned a lifetime factor of x10. Lifetime CO2 benefits are derived

from the table in section 5.1.2 of this report to ensure that the are consistent with

carbon budget figures.

As they are not quantifiable, additional benefits are not included.

Costs are derived from the total Defra spend on the programme for the 2010-11

budget year. Although there are costs to UK industry of implementing / complying

with new policies and initiatives, these are already accounted for within the ‗net‘

impacts. The table below shows the total programme spend for the Market

Transformation Programme in 2007-1115:

2007 - 8 £2.7m16

2008 - 9 £2.7m

2009 – 10 £2.7m

2010 – 11 £2.2m

2011 - 12 £1.9m

This makes a total contract spend – between 2007 and 2012 – of £12.2m.

In addition to the budgeted programme spend, for the last three years the person

cost for Defra of managing the programme has been approximately £260,000 per

annum17. However, as the Defra team also have other responsibilities, assessing

15 All £ figures are for the budget year April to March; spend for each year was roughly equally split

between six month periods. 16 Estimated (on the basis that the budget for 2007-8 also included production of SCP evidence) 17 Based upon Defra salary estimates.


Final report

April 2012

46

the precise cost to Defra of managing MTP would rely upon assumptions. Therefore

this cost has been excluded from cost effectiveness calculations.

6.8.2 MTP Value for money

Using the programme cost information, the following table sets out MTP value for money

for a scenario in which:

a. Only outputs where MTP have had a strong influence are taken into account

b. Any outputs where MTP have had at least some influence are taken into account

‗Policy vs. reference‘

‗Policy A vs. Policy B‘

Strong

influence

only

Lifetime savings

£1,484m and 7.1MtCO2 £348m and 1.61MtCO2

Impacts per £ spent

£122 and 0.58tCO2 £28.50 and 0.13tCO2

Strong

and

some

influence

Lifetime savings

£6,994m and 36.5MtCO2 £954m and 5.38MtCO218

Impacts per £ spent

£573 and 3tCO2 £78.20 and 0.44tCO2

Although the ‗policy A vs. policy B‘ figures are comparably smaller than for the ‗policy vs.

reference‘ figures:

These figures are more cautious and carry less potential for misinterpretation

These figures still show exceptional value for money, especially when it is

considered that unquantified impacts from efforts on Energy Star and Voluntary

Agreements have not been incorporated.

18 Figures for this cell take account of the fact that, for standby, attributed impact figures should not

have a lifetime factor applied because the impact was timing rather than stringency.


Final report

April 2012

47

7 Conclusions and recommendations

This section draws together the report findings around process, attributed impact and

wider benefits to inform our response to the evaluation objectives. This section seeks to

answer the following:

Is there value in the outputs being produced?

Are there opportunities to deliver these outputs – and their outcomes - in a more

efficient way?

Are there opportunities to deliver further benefits to the UK?

7.1 Is there value in the outputs?

At a UK, European and global level, MTP data and outputs have been used to influence

policy outcomes and programmes which carry substantial CO2 and consumer saving

impacts. The programme has also delivered good value for money when comparing the UK

benefits of policy outcomes influenced to programme budget.

The findings of this report provide strong evidence that – at a European policy making

level – the UK enjoys substantial influence upon the eventual policy outcome and that this

influence is underpinned by what the Commission and other Member States perceive to be

a robust evidence base.

The research suggests that development and possession of the types of output that MTP

produces have, amongst other benefits:

Given the UK a prominent role in pre-consultation scoping and a prominent

position at the European negotiating table;

Ensured a level of (almost automatic) support from some Member States for the

UK position;

Provided an important counter-balance to less progressive national / industry

interests;

Helped the UK to maintain some degree of control over the direction of policies

that could otherwise carry detrimental effects nationally;

Built good relations and reputation with stakeholders both domestically and

internationally;

Meant the UK does not have to use more political means of negotiating to a

desired outcome;

Helped to embed more effective monitoring and compliance.

For specific policy outcomes, there is a level of uncertainty as to the counterfactual – i.e.

what would have happened without MTP outputs – but overall it is clear that some policy

outcomes might have been very different without the UK taking an evidence-based

position.

A small number of stakeholders implied that other Member States can get by in

negotiations without the same support and that the UK – with a large number of votes –

could probably still enjoy some influence regardless of MTP. However, there are three

counter arguments to this view:


Final report

April 2012

48

1. Most respondents felt that a Member State would struggle to play an active role in

shaping policy without technical evidence bases and data.

2. Without MTP work, the UK could be less clear on which policy outcome would be in

the national / environmental interest and therefore what position to support.

3. The benefits of MTP outputs are not only realised in Commission negotiations, but

across a range of national and international initiatives and programmes.

With a number of Ecodesign and Energy Labelling implementing measures still to be

negotiated, ongoing development of UK procurement standards and voluntary agreements,

and an important role for programmes such as Energy Star and 4E, there is also ongoing

potential for the benefits derived from 2007-11 activities to continue to be realised.

Overall, it is therefore clear that there has been – and continue to be – value in

the UK working to collate and analyse data on energy using products and forecast

benefits.

7.2 Opportunities for more efficient delivery?

7.2.1 Why MTP?

Whilst most respondents agreed that the type of outputs produced by MTP are beneficial,

most other Member States do not have a separate government-funded programme to

deliver these. The alternatives – as demonstrated by other Member States – were to have

advisors from within government (often energy / environmental departments or agencies)

or industry.

With so many industry stakeholders expressing willingness to engage in data collation and

providing advice to government, and an enhanced Defra team - does the UK need to

deliver these outputs through a separate programme?

As another alternative, one phase 1 respondent felt that the programme outputs would be

better if Defra were to directly manage groups of experts and delegate specific technical

tasks to this group. This would allow more ad hoc resource use and potentially more

efficient delivery.

Alternative delivery methods are options that Defra may wish to consider. However, it is

also important to highlight that the research identified a number of arguments for

continuing to source the required data through MTP:

1. Even by the admission of some industry associations, useful data held and

provided by members is hard to find and inconsistent in quality. It would therefore

be a risk for the UK to rely upon such sources.

2. MTP is impartial and without vested interest in a developing data which supports a

particular outcome.

3. MTP is a recognised and respected brand at the European level. If MTP – or its

outputs – were simply to become a Defra-managed subsidiary which is called upon

for evidence for a regulatory position, the data – and so the UK – could lose some

of its reputation for impartiality and integrity.


Final report

April 2012

49

4. Through the contract, Defra have established a fairly consistent method of

specifying and obtaining the data required in the format required.

Overall, whilst this evaluation shows that there is value in both alternative

arrangements and retaining the current set up, it has certainly highlighted a

number of ways in which the current MTP process could work more efficiently

and effectively. These are discussed below.

7.2.2 Improving efficiency within the current framework

Although the value for money delivered by the programme is good, the research has

highlighted a number of ways in which the same benefits could be realised with less

resource:

Defra has invested heavily in the development and maintenance of the evidence

base and associated models which the consortium has developed. Now that the

architecture is set up, Defra should consider the extent to which it may wish to

further develop the models or alternatively consider setting contracts based on

outputs. In other words, Defra still need the outputs, but how much of the process

is now necessary?

The tendency in the current contract has been to develop outputs for any product

group under consideration. For some policy outcomes, it was felt by phase 2

respondents that the UK had effectively supported an outcome that would have

been very likely regardless of their input. In a similar vein, as indicated by the

exclusion of various policy outcomes for the phase 2 verification, there are

recognised to be occasions where the UK has not been able to influence policy

outcomes. This suggests a need for more strategic assessment of whether or not,

for a particular product group or policy:

a. The effects of changes to a product group upon the UK are of enough

significance to warrant full investigation and analysis of data.

b. The eventual policy outcome is likely to be affected by the UK position.

c. The eventual policy outcome is likely to be influenced by possession of robust

data.

When seeking to influence policies, Defra may wish to give more priority to those

outcomes and policies having the highest potential benefit for the UK. This in turn

could mean a more robust evidence base is required for some policy types within

MTP‘s modelling than others. It should be borne in mind that where there is at

least some impact of a policy / product group upon the UK, there will still be a

degree of analysis required for carbon budgeting purposes. However, the resources

devoted should be proportionate rather than uniform.

Whilst the current MTP structure focuses around the development of the evidence

and models, the Defra team have evolved to lead on particular product groups. It

is therefore worth considering whether the consortium could also be organised on

a product by product basis.

One of the key issues cited by MTP for turnaround of required outputs was the

management and availability of subcontractors. This is an issue common to


Final report

April 2012

50

subcontracting that perhaps should have been anticipated. Straightforward

solutions include a requirement in the contracts of independent experts to commit

to delivering work or making themselves available, or an expansion of the panel in

order to minimise the risk of expertise being unavailable for a particular product

group or policy area.

Although one Defra respondent acknowledged that efforts had been made to

identify savings that could be made in the MTP contract, one of the key issues in

the 2007 contract renewal was that the only bids competing with AEA for the

management role were not high quality19. Defra should work to encourage more

bidders in any new commissioning process in order to maximise the potential for

new and potentially more efficient approaches to be identified.

7.3 Enhancing programme effectiveness

7.3.1 Retaining work in key areas of impact

All phase 1 respondents saw the following MTP work-streams as key, and thought they

should be preserved in future contracts:

The Eco Design work stream was seen as crucial for influencing EuP legislation and

labelling at present and for the potential to influence ERP in the future. This is also

borne out by the phase 2 verification work.

International work, seen as important because it entails working with other big

markets such as Australia, the USA, China and Brazil enables Defra to influence big

manufacturers and drive up their standards. Examples cited included the SEAD

work and the Energy Star work.

Future studies, seen as important in working out likely scenarios for products and

product development and also on technical areas that can improve the evidence

base.

7.3.2 Enhancing the impact of current outputs

Drawing together both respondent views and our experience of using the policy

outcomes impact data, it is clear that although there is currently no modelling

system better placed to form an assessment of the impacts of the policies under

consideration, there is a need for enhancements to be made to the quality of the

assumptions and modelling used in the MTP evidence base. Furthermore, there are

a number of policies for which the outcomes and benefits at a UK level are unclear;

there would be benefit in seeking to evaluate these policies and quantify their

impacts more accurately.

It could be that more frequent revisions to the IAs may be necessary to update the

evidence bases used for modelling, particularly in fast-changing markets such as

consumer electronics and ICT. Clear markings of the updates and the significant

changes that have been made is also recommended.

19 Defra source.


Final report

April 2012

51

Something noted by several respondents in phase 1 was the lack of communication

MTP outputs and capabilities. It would be worthwhile for Defra to consider how

best to communicate MTP outputs and the audiences it wishes to communicate to

in order to improve this in future. For example, providing briefing notes to product

buyers might allow them to push for more efficient goods from their suppliers.

Some respondents suggested that the consortium should have an economist within

their team to help deliver Defra‘s expectations in this area. Closer working with

economics teams within Defra and maybe DECC was another useful

recommendation. Defra has recently seconded an economist to AEA.

Several industry and agency representatives in the UK suggested that more

information on where the MTP evidence base comes from would be useful to make

it easier to assess how robust and therefore how useful and reliable it is. They

welcomed the introduction of the risk matrices with published data and research.

MTP may wish to consider alternative formats in which the modelling and evidence

base information could be stored, or the potential for separate front and back-end

systems to allow the modelling results to be obtained quickly while retaining the

functionality of the modelling process. The ‗What-If‘ web modelling tool has been

removed from the MTP website. Whilst the number of regular users of this tool was

likely to have been small, the detail and capability that it provided may well have

led to positive impacts through use by other organisations or policy makers.

Three industry respondents suggested that more industry input could be useful to

MTP or Defra. They suggested that this could be a review by a small industry group

(either at the start of the study process or before publication) to ensure the

appropriateness of the scope or approach. They felt that this would provide a sense

check from the market view. Two of these respondents acknowledged that this

raised challenges in terms of resource and fairness that would have to be

considered and managed appropriately.

7.3.3 Future opportunities

Inevitably, if MTP resource is being focused upon policies where little or no influence is

being achieved, there may be opportunities being missed in other areas. The key

opportunities for the programme in 2012 and onwards are set out below:

Greater potential savings can be made through changes in EU policy; however,

Defra and MTP could be more likely to have an attributable influence on domestic

policy changes, including ‗beyond compliance‘ voluntary agreements and

procurement standards.

MTP should also consider where it may be best to achieve outcomes through

working with different parts of the product markets. MTP currently works on both

supply side (though minimum MEPS, encouraging innovation, and responsibility

deals) and demand side (through co-labelling, influencing consumer purchasing

behaviour). As part of considering outcomes, MTP should consider relative

priorities given to each type of work. Which has proven to be more/ less effective

through past activities?


Final report

April 2012

52

Overall, it is clear that MTP has evolved significantly over the period of the last

contract and so it is important to future-proof delivery wherever possible in the

future.

Defra need to assess what the likely changes are which will occur in UK, EU and

international policy in future and what impact these may have on the desired

direction for MTP.


Final report

April 2012

53


Final report

April 2012

54

8 Appendix A: Full methodology statement

Sections 8.1– 8.4 describe the overall approach taken to completing the different research

elements.

Section 8.5 and 8.6 describe the approach taken to establishing policy impact and

attribution of these to MTP.

8.1 Phase 1: desk research

The desk research focused on three types of document:

MTP programme inputs, including work plans and reports outlining the activities

the MTP programme had undertaken;

MTP outputs, including Briefing Notes and the ―Saving Energy Through Better

Products and Appliances‖ report;

Policy documents for areas which MTP has the potential to provide influence

leading to carbon impacts.

The table below shows the full list of documents included in the review, along with the

rationale for including each i.e. what information pertaining to the evaluation was obtained

from them:

Document Title(s) Rationale for inclusion in review

MTP Work Plan 2010/11 – Issue 4

Outlined activities undertaken by MTP and the spend allocated to those activities

MTP Work Plan 2011/12 – Issue 3

MTP - Proposed budget for 2011/12

Provided detail on the resources allocated to different workstreams

MTP Organogram Set out the MTP management structure and lines of reporting

Carbon Plan Action Summary Set out MTP's role within the context of the Carbon Budget

UK Carbon Plan Provided understanding of the policy context for MTP priorities and activities

IEA 4E Mapping and Benchmarking Annex

Provided information on this exercise and MTP‘s input to it

Defra's structural reform plan Explained how wider Defra spending and reform has affected MTP

MTP Quarterly Reports (Quarter 1, 2010/11 - Quarter 1, 2011/12)

Outlined activities undertaken by MTP and the spend allocated to those activities

MTP Evidence Base Progress Update Brief November 2010

Stated the progress of key MTP Evidence Base workstream tasks up to the end of November 2010


Final report

April 2012

55

Modelling the energy use of products - A review of approaches from practice

Identified potential options for AEA to improve the cost effectiveness of the MTP modelling approach

MTP Evidence Base - Future Strategy

MTP EuP Monthly report – July 2011

Showed progress against budgets and deliverables for the EuP workstream against each project's original aims Reports for Energy using Products

workstream June 2009 - January 2010

Net Benefits from MTP IAs and latest departmental figures

Provided Defra and MTP views on the extent to which Defra has influenced EU policy and outcomes. The ‗SEUP Benefits‘

Defra contribution estimates enabled the ‗Policy A vs. Policy B‘ comparison.

Internal MTP ‗SEUP Benefits‘ document used as a guide to Defra‘s individual contribution to benefits delivered by EU Regulations.

Energy and CO2 savings for EuP

8.2 Phase 1: Consortium interviews

Interviews were undertaken with Defra and MTP consortia staff, including individuals who

were previously closely involved in the MTP programme but are no longer. These

interviews were designed to gather information on:

Programme delivery structures;

MTP operational success and challenges;

Potential areas of impact for exploration in phase 2.

In total, fourteen qualitative interviews were undertaken; these were split as follows:

Defra – current – 6

Defra – previous – 2

AEA – current – 4

AEA – previous – 2

Each interview was tailored to explore issues pertinent to the respondent‘s interaction with

MTP. The full discussion guides used for the consortia interviews are shown in the

appendices to this report.

The key risk of the approach was that MTP and Defra may (though this is unlikely) have

been unaware of influencing a particular policy outcome e.g. where the evidence base has

been utilised without direct communication with MTP. Were this the case, such outcomes –

and their benefits - would not have been explored in phase 2 and would not be accounted

for in value for money analysis.

8.3 Consortia workshop

Following completion of the desk research and consortia interviews, a review workshop

was facilitated by Databuild at Defra offices. This was to enable discussion of:


Final report

April 2012

56

Summary findings and areas for consideration in the specification of the next

contract, which Defra were in the process of developing.

Areas of MTP outcomes and impacts that should be explored in phase 2.

8.4 Phase 2: Stakeholder interviews

Phase 2 of the evaluation aimed to provide verification of – and further insight into – the

areas of impact identified in phase 1.

8.4.1 Impacts explored

The desk review, consortium partner interviews and Defra / AEA workshop identified a

number of areas where the MTP programme was likely to have had an impact:

EU Eco-Design of Energy Related Products Directive:

- Ensuring consideration of particular products within the directive programme

- Influencing an increase in the agreed minimum energy performance standard

(MEPS) required for a particular product group

- Influencing the quicker introduction of minimum performance standards

- Advising on testing and compliance to enforce the new standards

Influencing the design of labelling of energy related products as part of the

mandatory EU labelling scheme

Influencing voluntary agreements with industry; specifically the phase out of GLS

bulbs and introduction of lower standby voltages for appliances.

Green procurement: influencing the level of energy performance standards

required for products purchased through government contracts.

International programmes; influencing the requirements of Energy Star, the public

procurement standard for all US and EU countries.

8.4.2 Stakeholder interviews conducted

Based upon these areas where Defra and MTP believed the programme had influenced

outcomes, Databuild worked with Defra and MTP to build a database of individuals who

would be appropriately placed to provide verification that this was the case. The chart

below shows which stakeholder groups were contacted in phase 2 and what areas they

were expected to provide verification on:

Stakeholder groups Interviews conducted Impact areas covered

Defra product / policy leads 4

MEPS and labelling,

voluntary agreements,

compliance, green

procurement.

MTP consortia leads 3 MEPS and labelling,

voluntary agreements.

European Commission policy

leads 7 MEPS and labelling.

Individuals representing

other EU member states at

directive negotiations

5 MEPS and labelling.


Final report

April 2012

57

Industry representatives 9 MEPS and labelling,

voluntary agreements.

Energy Star programme

representatives 2

International programmes,

compliance

NMO 2 Compliance

8.4.3 Challenges encountered in securing interviews in phase 2

1. Sample coverage of product groups and policies – some respondents were able to

comment on a number of product areas or on more than one policy instrument;

but for some policy instrument / product group areas, appropriate respondents at

the European level could not be identified. The product groups for which responses

were primarily lacking were EPSUs, electric motors and circulators.

2. Ability / willingness to participate – six potential interviewees identified in

consultation with Defra were either unwilling to participate or could not be

contacted. This was a product of the high levels of movement of staff identified

through this research. The consequence of this was that responses could not be

obtained from some of those best placed to provide verification.

8.5 Impact methodology

8.5.1 What impact data was required and why?

In order to ascertain the outcomes of MTP activities (in terms of £s or CO2), the impacts of

the policies and programmes they intend to influence also had to be determined.

The decision was taken to focus upon UK impact because:

a. This is the basis upon which most of the impact assessments and forecasting

produced by MTP have been calculated. Although indicative extrapolations of the

UK impacts to European level have been conducted, these were not intended to be

robust. Instead they were taken to be indicative of the potential level of influence

MTP could have at the European level.

b. This provides the appropriate basis to assess the value of spending UK public funds

i.e. impacts in Europe and / or beyond do not (at least directly) produce

environmental or financial benefits for the UK.

The sections below set out the approach taken in the evaluation to quantify the impacts of

the different policy instruments and programmes on the UK.

8.5.2 Sources of Impact Data

Latest carbon budget impact figures (derived from MTP IAs and the DECC Carbon Plan)

were used as the most robust monetary and carbon impact figures available for MTP‘s

work.


Final report

April 2012

58

The document ‗Saving Energy Through Better Products and Appliances‘ gives some detail

on the way in which MTP‘s Impact Assessments use modelling to assess the likely impact

of new policies: “The modelling approach used is mainly based upon a stock turnover

model, where the number of products in use, expected growth or decline of that number in

future years and product lifetime form the basis for calculating how the installed stock in

any one year is composed of products sold in current and previous years. This data provide

the basis for analysis and scenario building to quantify the current and future

environmental performance of products, and the likely effects of various policies and

changes in the market. This analysis allows the Government to assess the expected impact

of different policy options over time and decide on adequate performance standards for

energy-using products.”

In transferring the Impact Assessment (IA) impacts into the carbon budget figures, some

alterations to the calculation mechanisms were made in order to ensure that they were up-

to-date and consistent.

1. Monetised benefits such as the ‗total value of absolute reduction in required

delivery of renewable energy‘ were removed. This was included in the analysis

used for the Impact Assessments as a requirement of the DECC Guidance on

Appraising Energy and GHG Policies, but was taken out of scope for the purposes

of the departmental figures. The change ensures that the benefits the UK enjoys as

a result of needing to generate less renewable energy are not over-stated in

different Impact Assessments.

2. Some assumptions used in the impact assessments were updated to reflect the

time that the departmental figures were produced e.g. the base price year. This

change ensures that up-to-date assumptions about product prices and costs were

used when generating the impact figures.

The list below shows which Impact Assessments were used to generate impact figures for

each product group in the departmental figures:

Implementing

Measure

MTP Impact Assessment(s) included in this product

group

Standby Impact Assessment of EuP Implementing Measures for

Standby and Off-Mode Losses Version 1.0

ESPUs Impact Assessment of EuP Implementing Measure for External

Power Supplies Version 1.0

Electric Motors Impact Assessment of EuP Implementing Measure for Motors

Version 2.0

Circulators Impact Assessment of EuP Implementing Measure for

Circulators Version 7.2

Cold appliances and

labelling

Impact Assessment of EuP Implementing Measures for cold

domestic appliances version 1.8

TVs and labelling

Impact Assessment of Implementing Measures for eco-design

requirements and energy labelling of Televisions (TVs) Version

4.7

Wet appliances and

labelling

Impact Assessment of EuP Implementing Measures for

washing machines Version 1.3

Impact Assessment of EuP Implementing Measures for


Final report

April 2012

59

dishwashers Version 5.0

Domestic Lighting Impact Assessment of EuP Implementing Measures of

Domestic Lighting Version 2.4

Tertiary Lighting Impact Assessment of EuP Implementing Measures for Tertiary

Lighting Version 1.0

Simple STBs Impact Assessment of EuP Implementing Measures for Simple

Set Top Boxes Version 1.0

8.5.3 Overlap with other policies

In some product areas, more than one policy or initiative was identified as being

potentially supported by MTP. In these cases, impact figures should not be calculated as

the likely impact of policy A plus the likely impact of policy B, as the two policies are likely

to have influenced some of the same impacts (in other words, some impact achieved by

one of the policies is the same impact that would have been achieved by the other). The

impact assessments of EuP policies take account of this by including other policies which

may affect the product group in the reference scenario. This ensures that the impact

figures calculated relate only to the EuP policy in question.

Based upon review of the impact assessment documents, the table below shows, for each

product group considered in the evaluation, which policies have been accounted for when

calculating the overall impact:

Policies considered in IA analysis

EuP Directive

Energy Labelling

ENERGY STAR CERT

Building Regs ECA CRC Others

Pro

du

ct

Are

a

Standby

ESPUs

Electric Motors

20

Circulators

Cold appliances and labelling

TVs and labelling

Wet appliances and labelling

Domestic Lighting

21

Tertiary Lighting

Simple STBs

Key:

- Included in IA analysis

- Considered for inclusion in analysis but effect deemed negligible

20 Government Quick Wins Scheme 21 UK Voluntary Agreement


Final report

April 2012

60

8.5.4 Impact data quality and limitations

The limitations of the IA data are described in detail in the main body of the report (section

4).

Of these, one of the most important was that the IA figures assess the impact of the MEPS

/ labelling policy compared to a reference scenario of there being no policy. In reality, this

reference scenario would be very unlikely and reporting MTP influence against these

impacts risks misinterpretation of the actual extent of impact MTP could reasonably have

influence.

For example, for standby, MTP data was predominantly used in order to secure the

introduction of more stringent standards 1-2 years earlier than would otherwise have been

the case. A MEPS policy would have been introduced regardless and more stringent

standards would almost certainly have been in place by the third year from the

introduction of the MEPS. It is therefore misleading to suggest that MTP influenced

anything more than the marginal impacts derived from two years of having more stringent

requirements, as the vast majority of the lifetime savings from the policy would have been

delivered anyway.

These limitations have necessitated – for completeness - exploring MTP influence upon two

different types of impact:

1. The impact of the policy compared to not having a policy at all (‗policy vs.

reference).

2. The impact of the MTP-influenced policy compared to the impact of the likely policy

outcome if MTP had not been involved (‗policy A vs. policy B‘)

It is important to understand the risks in using the larger ‗policy vs. reference‘ impacts

rather than the ‗policy outcome A vs. policy outcome B‘ impacts.


Final report

April 2012

61

8.6 Attribution methodology

The attribution methodology in this section explains how MTP‘s influence upon particular

policy instruments and product groups were identified and verified. It also explains the

rationale behind the way in which attribution has been reported and integrated with impact

data in this report.

8.6.1 Stage 1: identifying potential areas of MTP influence

The activities of MTP are numerous and wide-ranging. It was therefore agreed that the

most efficient use of evaluation resource would be to use the desk research and consortia

interviews conducted in phase 1 to filter a set of outcomes where Defra and MTP were

confident that impact had been achieved.

Desk research and the phase 1 consortia interviews identified the main points at which

Defra, and the MTP consortium, have influenced policy making22. For each priority area

Defra and MTP identified a number of specific and general claims on what had been

achieved and how it had been influenced.

The desk review, consortium partner interviews and Defra / AEA workshop identified a

number of areas where the MTP programme was likely to have had an impact:

EU Eco-Design of Energy Related Products Directive:

- Ensuring consideration of particular products within the directive programme

- Influencing an increase in the agreed minimum energy performance standard

(MEPS) required for a particular product group

- Influencing the quicker introduction of minimum performance standards

- Advising on testing and compliance to enforce the new standards

Influencing the re-design and re-valorisation of labelling of energy related products

as part of the mandatory EU labelling scheme

Influencing voluntary agreements with industry; specifically the phase out of GLS

bulbs and introduction of lower standby voltages for appliances.

Green procurement: influencing the level of energy performance standard required

for products purchased through government contracts.

International programmes; influencing the requirements of Energy Star, the public

procurement standard for all US and EU countries.

8.6.2 Stage 2: verifying the claimed influences

Following identification of the outcomes that would be focused on, phase 2 stakeholder

interviews were undertaken with the aim of verifying, or otherwise, the general and

detailed list of specific MTP impacts.

22 Note this does not include some areas such as future studies and international, which

can take the form of different studies throughout the year.


Final report

April 2012

62

8.6.3 Challenges in verifying MTP influence

There were a number of challenges faced when attempting to verify the hypothesised

influence of MTP; these included:

Movement of key personnel within the Commission; this meant that the desk

officers with first hand or most pertinent experience of MTP input to policies over

the past 3-4 years were not available (or in two cases not willing) to participate

and provide feedback. The other effect of this was that some of the recommended

respondents did not feel that they had been in their posts long enough to be able

to comment fully on the influence of Defra or MTP on the policies for which we

have estimated an impact.

No direct contact with MTP; although selected on the basis that they had some

experience of working with MTP and / or experience of the policies being

researched, some respondents had not had direct contact with MTP or were

unaware that MTP was responsible for a particular piece of analysis or technical

input to Defra. Similarly, some UK trade bodies input to the Commission legislative

process through a European level trade federation. Therefore some industry

respondents did not have sufficiently direct involvement in the legislative process

to be able to directly assess the effectiveness of Defra or MTP in influencing the

outcome in those negotiations.

Complexity of the decision making process; the legislative process at the European

level is complex with a large number of participants and multiple stages. The

process is one of building consensus and negotiation, involving multiple and often

differing opinions, taking into account both political and technical arguments.

Many respondents therefore found it difficult - or were unwilling - to attribute

outcomes to one particular stakeholder or one particular contribution. In a similar

vein, several respondents at the Commission felt they were unable to attribute

influence to an individual Member State or comment on specific events in the

legislative process on the basis of confidentiality linked to the commitology

procedure. There was the sense that some member states respondents were not

willing to go too far in crediting another individual Member State with influencing

an outcome.

8.6.4 Stage 3: Arriving at an assessment of attribution

On the basis of the challenges outlined above, the following were assessed in order to

provide categorisation of attribution for each policy outcome:

1. The extent to which Defra/MTP had influenced the outcome of the policy making

process, both generally and for specific claimed impacts. The level of influence was

rated as follows:

Strong influence - MTP (alone or through Defra) were critical in influencing

the outcome;

Some influence – MTP (alone or through Defra) made some contribution to

the outcome ;

Weak influence – MTP (alone or through Defra) made a slight difference to

the outcome;

No influence - MTP (alone or through Defra) made no difference to the

outcome;


Final report

April 2012

63

2. The level of confidence in that assessment of influence; established through a

‗traffic light‘ system:

Green – there is good verification of the Defra / MTP claim, with at least

one respondent is able to authoritatively verify the claim and there is a

good match in the detail for specific claims.

Amber – there is some verification of the claim; at least one respondent is

able to verify the claim but there is some uncertainty in the recall of the

detail of specific claims.

Red – there is no verification of the claim made by MTP / Defra or there is

disagreement between respondents or such a large degree of uncertainty

that verification of the actual influence on the outcome is difficult to

discern.

In all cases of assessing the level and reliability of the attribution categorisation, the

following were taken into account across applicable responses:

The varying degrees of respondent involvement in the negotiation process;

The varying degrees of political interests / vested interests that might affect

responses.

8.6.5 Stage 4: Integrating impact data with attribution data

Regarding MEPS (as the focus of most MTP work), prior to the evaluation, consideration

had already been given to MTP‘s influence on the policy outcomes relating to the product

groups being considered under the directive. Defra had undertaken a brief review which

resulted in a rough estimate of the proportion of the SEUP benefits impacts that would not

have been achieved without MTP.

It was originally anticipated that these % attributed impact ranges could be used as a

starting point to test with phase 2 respondents to draw out estimations of % attribution.

However, this was not done as Defra expressed reservations about the potential for

stakeholders to misconstrue the figures as robust and evidence-based claims.

There were therefore two ways in which the attribution and impact data could have been

integrated:

1. Work to estimate other precise % influences to enable calculation of

impacts delivered by MTP activities

This would have entailed ascertaining a description of the influence MTP activities

had upon a particular policy outcome, and then ascribing a proportion of the

impact of that policy outcome to MTP i.e. assuming that precise amount of impact

would not have been achieved without MTP input.

The main flaw with this approach as regard the MTP evaluation is that the ways in

which MTP may have influenced a particular outcome could be both nuanced and

numerous. Therefore arbitrarily selecting a proportion of impact to claim for MTP

ignores the issues around what the true counterfactual would have been and issues

around how the SEUP benefits impact data might have been disaggregated across

different levels of stringency / time delay etc.


Final report

April 2012

64

2. Stating the impacts which MTP has influenced to some degree

This approach simply takes the ‗policy vs. reference‘ and ‗policy A vs. policy B‘

impacts - which MTP has attempted to influence - and states the extent to which

MTP has had an influence upon that impact. The limitation of this approach is that

a precise impact is not ascribed to MTP influence.


Final report

April 2012

65


Final report

April 2012

66

Results

Evaluation of the Market Transformation Programme Final report

Documents

Transcript of Evaluation of the Market Transformation Programme Final report