01/10/2015

1

Topic 2.Programme Evaluation:

key Qs, methodologies & guidelines

Dr Micheál Collinsmlcollin@tcd.ie

1. Why Evaluate?2. The Public Spending Code3. Appraisal/Evaluation Glossary4. Life-Cycle of a Programme/Project5. Guidelines & Requirements6. Appraising Current Expenditure7. Value for Money Studies8. Other Notes9. Discussion10. Reading for Topic 2

1. Why Evaluate?

Limited resources; unlimited uses

Choices bases on systematic procedures

Aim for ‘efficient’ use of resources min inputs for a given output where output is clearly defined

Answer the questions under five headings/purposes:

01/10/2015

2

1) Planning Is the programme justified? Input into programme design Efficient resource allocation

2) Implementation Is the programme working and/or how can the

programme be improved?

3) Accountability What was achieved?

4) Knowledge & Learning What interventions work and in what circumstances? Does the logic of the programme and its assumptions

need to be questioned? Lessons for other applications

5) Development Institutional performance and strengthening Service quality

But…

evaluation sometimes used for other, “covert” purposes

Justify decisions already made Postpone decisions Public relations Compliance

ideal?

“A rational exercise often undertaken for non-rational reasons” (Weiss)

“To learn through systematic enquiry how to better design, implement and deliver public programmes and policies” (EU EVALSED Guide)

01/10/2015

3

2. The Public Spending Code

Irish context: recent notable change Previously: range of various documents, guidelines and

circulars problematic and inaccessible reform Publication of ‘The Public Spending Code’ by DPER Housed under the CEEU

Central Expenditure Evaluation Unit Input into this week and next Underpins methods & guidelines Plans to continue to update

An input into decision making

An introduction to the PSC: (‘general points’) Building on Good Practice

brings all the past documents together allows easier access, application and update ‘a responsive and evolving code’

Aid to good decision making formalised the input into this process but…(see later)

Proportionality scale of appraisal or evaluation should be proportionate to

the scale of what is being reviewed some points/guidelines later

Appraisal never to be case-making role to be objective rather than campaigning

Avoiding premature commitments When undertaking one, care not to represent it as a

commitment to do / retain something it is a commitment to consider not implement / keep be prepare to abandon proposal/programme is appropriate:

EU Funding Ireland moving to be a net contributor ‘a national resource’ …used in the same way as own

revenue ‘The availability of EU aid for a project is not a justification

for investment in that project’ often parallel EU evaluation requirements (see EVALSED)

01/10/2015

4

Code stresses how evaluation = an input into decision making rather than decision making:

3. Appraisal/Evaluation Glossary

Lingo…useful to define now Appraisal/Evaluation

interchangeable before/after

Sponsoring Agency Government Department, other state body

Sanctioning Authority Usually the provide of the € Government Minister or Department Cabinet above €100m (was €250m until Sept 2015)

4. Life-Cycle of a Programme/Project

Irish guide: 4 stages UK guide: 6 stages

1. Appraisal – before expenditure incurred preliminary appraisal if €5m+

statement of needs (why do it?) objective and degree these would be met alternative options indication of costs

review and decide if more detailed appraisal needed if appropriate undertake detailed appraisal

involves: clarification of objectives, exploration of options, quantification of costs and a method of selecting the best solution from competing options

01/10/2015

5

2. Planning/Design Appraisal approval costing and planning more on method/steps coming up…diagram

3. Implementation internal or contracts difference between time-defined and ongoing programmes

4. Post Implementation Review May be once established or once completed or both judge versus objectives lessons to learn (where express these?)

01/10/2015

6

5. Guidelines & Requirements

Scale of Appraisal Picking up on the proportionate point earlier € have moved around over time current ones:

€ Range Appraisal

< €0.5m ‘a simple assessment’…discuss

€0.5m-€5m ‘a single appraisal involving elements of preliminary and detailed appraisal’

€5m-€20m Multi-Criteria Analysis (MCA)

€20mAnnual €5m+

Current: economic appraisalCapital: CBA or CEA

Annual €30m5yrs+

Prior and mid-term evaluation

01/10/2015

7

6. Appraising Current Expenditure Programme evaluation: 5 key questions

Reflects DoF Value for Money manual (2008) Originally from CSF Evaluation Unit (1996)

1. Rationale2. Relevance3. Effectiveness4. Efficiency5. Impact

Looking at each of these… Relevant to reviews of existing programmes and

considerations of new ones.

Rationale What is the justification or rationale for the policies

underpinning the programme? What is the underlying market failure justification for

Government intervention?

“A necessary but not sufficient for government intervention to improve economic efficiency is that there is some form of market failure” (HM Treasury)

Could the problem be addressed through more direct means?

Danger of “second-best” solutions

Relevance What are the implications for the programme of changes

in the wider socio-economic environment and in the context of overall Government policy? What societal needs or problems does programme

address? (needs assessment) Is the programme relevant given current priorities? Is a current programme relevant give current

situation and changes since it was introduced (fit-for purpose)?

Implications of external changes for programme (continued relevance)

01/10/2015

8

Effectiveness Is the programme meeting its financial and physical

objectives? Generally addressed at level of: Inputs: Is the money being spent? Outputs: Results or immediate benefits

Are they in line with expectations? If not, why not?

Similar to a monitoring question Correlation to assessments of impact (later) Dependence on how the objectives have been framed

Measuring effectiveness, challenges:

Effectiveness in the private sector objectives normally clear:

improve profitability…(benchmark measure) gain mkt share increase share price / company value

Effectiveness in the public sector objectives are difficult to specify why…

Difficulties in specifying effectiveness in the public sector People are unsure if asked Broad objectives – difficult to pin down Can be complex – improve health, provide educ. Often multiple objectives Easy versus difficult Conflicting objectives – economic v social Objectives can change Different conceptions – education Unplanned effects

01/10/2015

9

Efficiency Could more be achieved for the resources invested? Considered a core question…the best return for a given

amount of inputs Issues such as:

is it possible to get reduced inputs for same level of service

is it possible to get additional outputs for same level of inputs

is it possible to get improved unit cost ratio could you change the mix of activities/outputs to

better deliver a given objective for same input level could you use alternative delivery approaches, e.g.,

outsourcing to private sector

Impact What socio-economic changes can be attributed to the

programme? to beneficiaries to society generally Need to take account of:

Deadweight / displacement any unintended side-effects

Objectives of programme (if property defined) will be of help.

Most projects will be considered in the context of a sponsoring agency’s business plan or a multi-annual investment programme

Timing impacts on ability to answer these questions

Nice summary, ‘the evaluation cycle’ (David Hegarty, DOF)

01/10/2015

10

7. Value for Money Studies 2007 VFM manual (website) Details in the DPER documents (website) Value for Money Policy Reviews (VFMPRs) Really focussed on assessing the earlier questions

economy efficiency effectiveness the three Es

DPER/CEEU attempting to establish a standardised approach

Structure/Table of contents recommended by DOF:

6.2. Content of a VFM Review Report (from DOF guidelines) Title or Cover Page; Table of Contents; Glossary and Abbreviations/ Acronyms; Executive Summary; Introductory Chapter; Chapters reporting the findings of the VFM review , for

example: objectives of Programme X and the Rationale for the programme; inputs, outputs and the trend in efficiency achieved; effectiveness; impact; continued relevance and alternative means of achieving

Programme X objectives; and potential future performance indicators; and

a Final Chapter with Conclusions and recommendations

Annexes, to include: TOR Summary Sheet; the programme logic model template; members of the Steering Committee and evaluation team; details of surveys undertaken; review of international practice; major points raised by Quality Assessor and response by

Steering Committee; and the final two Annexes should separately list the persons

interviewed and the documents reviewed for the review.

01/10/2015

11

Programme Logic Model: maps out the shape and logical linkages of a

programme. provides a systematic and visual way to present and

share understanding of the cause-effect relationships between inputs, activities, outputs and outcomes (results and impacts)

01/10/2015

12

Answering the 5 key questions:1. Rationale2. Relevance3. Effectiveness4. Efficiency5. Impact

See notes attached to slides from VFM manual (p32-36)

8. Other Notes Links to Project evaluation – next week Role, as mentioned earlier, for both:

Post-project evaluations Periodic evaluations

Missing: more on financial analysis over next two weeks

9. Discussion

01/10/2015

13

10. Reading for Topic 2 Weiss article – see website Department of Finance (2007) VFM manual – see website DPER The Public Spending Code

http://publicspendingcode.per.gov.ie/ European Union (2012) EVALSED :The resource for the

evaluation of Socio-Economic Development - Evaluation guide. http://ec.europa.eu/regional_policy/information/evaluations/guidance_en.cfm

Scottish Government Guide: http://www.scotland.gov.uk/Topics/Government/Finance/spfm/appraisal

V A L U E F O R M O N E Y A N D P O L I C Y R E V I E W I N I T I A T I V E G U I D A N C E M A N U A L

32

3.4. Evaluation criteria

3.4.1. Evaluation criteria for VFM Reviews The evaluation criteria are reflected in the specific questions that the evaluation will have to answer. In the case of a VFM review, the specific questions are set out in the template Terms of Reference and cover the evaluation criteria: rationale, efficiency, effectiveness, impact and continued relevance. The criteria encompass the established Value for Money criteria (economy, efficiency and effectiveness – note that VFM reviews address ‘economy’ under the ‘efficiency’ heading). Other public policy evaluation criteria may also be relevant depending on the nature of the review (See Section 3.5). Each of the five evaluation criteria (rationale, efficiency, effectiveness, impact and continued relevance) is discussed in the following paragraphs.

3.4.2. Rationale Rationale is concerned with establishing why a public policy intervention is necessary in a given area. It requires consideration of the public policy objectives of a programme and the reasons for public sector provision or involvement. It is closely linked to the economic concept of market failure. Market failure is considered in more detail in Annex 1 but in simple terms, it exists where private individuals or firms do not produce the optimal level of a good or service from a societal perspective. A practical example of market failure is the need for subsidised bus services on socially desirable yet uneconomic routes.

The rationale criterion usually also extends to whether the design of the programme or project is the most appropriate means of achieving the identified public policy objectives.

In a VFM review, the study of rationale covers the following issues: Rationale

Issues Performance questions that can

arise Indicators

The validity of programme objectives

Why is a public policy intervention necessary?

What market failure is being addressed?

Are the programme objectives valid?

Are the programme objectives compatible with objectives in the statement of strategy?

How does the programme fit-in with other programmes of the Department or programmes of other Departments/Agencies that target the same category of beneficiary or economic/social issue?

Is the programme rationale kept under review?

Reasons for public policy intervention

Evidence of market failure

Whether the programme objectives were valid to start with and whether they are kept up to date

Documented links to statement of strategy

Linkages and dependencies between programmes at the expected results level

The frequency of review

V A L U E F O R M O N E Y A N D P O L I C Y R E V I E W I N I T I A T I V E G U I D A N C E M A N U A L

33

3.4.3. Efficiency and economy Efficiency and economy look at performance in terms of the production of outputs and the acquisition and use of inputs (See table below). It is common for economy and efficiency to be studied together in an evaluation report although, in many cases, it is also worthwhile to study them separately. Definition Programme Logic

Model

Economy Securing the appropriate quality of inputs at the best price

Inputs

Efficiency Optimising the ratios of inputs to outputs Inputs, Activities, and Outputs

Efficiency is a comparison of outputs to inputs and can be viewed in two different ways:

• whether the same level of output and result could be achieved with fewer inputs, i.e. at a lower cost; or

• whether a higher quality or quantity of outputs might be delivered from a fixed amount of input.

Evaluations of economy are in-depth considerations of inputs. The template VFM review Terms of Reference (see Annex 2 to Chapter 2) include a separate criterion for a study of the trend of resource inputs.

A study of efficiency and economy will cover some or all of the following issues. Performance questions that can

arise Indicators

Efficiency Issues

The throughput of outputs

What are the outputs? How many are produced?

Are output targets met?

Actual number of outputs produced

The timeliness of production of outputs

How long does it take to produce an output?

Are time targets met?

What is the backlog of production in the system? Is it getting better or worse?

Outputs produced per hour, day, week or month

Average time to produce a unit of output

Movement in backlogs

The Quality of the Output

How complete and accurate are outputs?

How many complaints are there about outputs?

Accuracy level (e.g. of a report)

Number of valid complaints about outputs

The Cost of an Output

How much did the output cost – per item and altogether?

Distinguish between programme and administration cost.

Unit cost of output

Management and resource allocation

Are the management structures and allocated resources appropriate for the achievement of programme objectives?

Is the measurement framework (indicators) adequate?

Management arrangements

Budget allocations

Proposed monitoring indicators and targets

Economy Issues

The cost of the resource inputs

Was the best price obtained?

Could a cheaper price be obtained? E.g. performing work during normal hours instead of on overtime

Actual cost of input

Unit price or cost of input

V A L U E F O R M O N E Y A N D P O L I C Y R E V I E W I N I T I A T I V E G U I D A N C E M A N U A L

34

Identification of wasted expenditure – inputs paid for but unused

How do the inputs compare with the inputs of equivalent programmes elsewhere?

Cost of unused inputs

Relative cost of inputs compared

The quality or mix of the inputs

Was a higher and more expensive grade of staff used? E.g. an AP doing the work of a HEO

Was the quality of inputs maintained? E.g. through staff training

Actual number of staff days input

Ratio of grades of staff used

Percentage of payroll spent on training

3.4.4. Effectiveness

Effectiveness is defined in terms of the extent to which the objectives have been achieved and the planned benefits delivered. (Achievement of impact objectives is considered under the impact heading.) It should be kept in mind that the results can be positive or negative, and intended or unintended. In terms of the programme logic model, the study of effectiveness is a study of the relationship between outputs and results. Effectiveness will consider the immediate to short term effects that the availability of outputs had on the targeted beneficiaries.

In a VFM review, the study of effectiveness should include the following issues: Effectiveness

Issues Performance questions that can

arise Indicators

The extent that the objectives were achieved?

What did the programme achieve?

Can the “performance gap” be measured or otherwise defined?

What are the contributing factors to the performance gap?

Is there satisfaction with the level of achievement of results?

Quantitative and qualitative measures of actual and expected performance

Input/output indicators that reflect contributing factors

The way that the achievement of objectives is assessed

What methods are used to assess effectiveness?

Is the achievement measured against a valid baseline?

Is the feedback on effectiveness used in setting new objectives?

Appropriateness of indicators used

Suitability and accuracy of baseline indicators

Evidence of assessments of effectiveness informing policy development and objective setting

Cost effectiveness

Could the same results be achieved in a better or cheaper way?

Benchmark comparisons to other approaches to the same problem nationally and internationally

V A L U E F O R M O N E Y A N D P O L I C Y R E V I E W I N I T I A T I V E G U I D A N C E M A N U A L

35

3.4.5. Impact

Impact is concerned with the wider effects of the programme for a larger group of persons (e.g. a sector) or for society as a whole (it includes the medium to long term effects on targeted beneficiaries); the successes and failures in achieving the wider objectives; and the main causes of this. In the study of impact, the same basic performance questions used for effectiveness are applied to the medium and long term sectoral objectives of policy interventions and programmes. The impact of a programme is the difference it has made. As with effectiveness, it should be kept in mind that impacts can be positive or negative, and intended or unintended

The consideration of impact extends the evaluation of effectiveness into the wider sectoral or national level. The issues that should be included in impact assessment are: Impact Issues Performance questions that can

arise Indicators

Socio- economic objectives

What are the longer-term impacts on targeted beneficiaries?

What are the wider socio-economic effects of the programme?

Indicators directly related to the targeted long-term impacts on direct beneficiaries

Relevant socio-economic indicators

Causality Is it possible to isolate the programme contribution to wider impact?

Were there unintended side effects?

Direct contribution of programme to strategic objectives

Measures of additionality, deadweight and displacement

Other public policy evaluation criteria

What are the other relevant policy impacts?

Poverty impact assessment

Equality impact assessment

Regulatory impact analysis

Rural proofing

The definitions of economy, efficiency, effectiveness and impact always need to be customised to the inputs, activities, outputs, results and impact of the Department or programme being studied. This is why it is important to produce a programme logic model as part of the evaluation planning papers for all VFM reviews. (See Chapter 4)

3.4.6. Continued relevance In a VFM review, the evaluation of the continued relevance of the programme covers the following issues:

Continued Relevance

Issues

Performance questions that can arise

Indicators

Continued relevance

What is the justification for the continued allocation of public funds to the programme?

How does the programme fit in with other programmes of (i) the Department, (ii) other Departments, and/or (iii) other public bodies in delivering on the Government’s overall policy objectives in the area in which the programme operates

Estimate of programme benefits expressed in quantitative, qualitative and timing terms and taking account of what would have been achieved anyway in the absence of the programme.

Changes in programme context/environment.

V A L U E F O R M O N E Y A N D P O L I C Y R E V I E W I N I T I A T I V E G U I D A N C E M A N U A L

36

Has the examination of effectiveness shown that the programme is achieving its objectives?

The five evaluation criteria (in green) can be mapped to the programme logic model (in red) in the following way.

3.5. Additional public policy evaluation criteria In addition to the five main evaluation criteria outlined above, VFM reviews will need to take into account a range of additional public policy criteria that are relevant to the design and implementation of Government programmes. These are sometimes referred to as horizontal principles. The most important are addressed briefly below, with references for further reading given in each case.

Depending on the nature of the programme being reviewed, it may be appropriate to address the additional public policy criteria under one or more of the five evaluation criteria outlined in section 3.4.1 above. For example, for a programme run by the Department of Social and Family Affairs, poverty impact assessment might be handled under Effectiveness and Rationale. Alternatively, they may be handled as separate criteria in their own right.

Further guidance on the development of evaluation criteria in the planning of VFM reviews is set out in Chapter 4.

3.5.1. Poverty Impact Assessment Poverty Impact Assessment is defined as the process by which Government Departments, local authorities and State agencies assess policies and programmes at design, implementation and review stages in relation to the likely impact that they will have or have had on poverty and on inequalities which are likely to lead to poverty, with a view to poverty reduction.

Under the Poverty Impact Assessment guidelines issued by the Office for Social Inclusion (OSI) , Poverty Impact Assessment should be conducted prior to finalising any Memorandum for Government and in the preparation of SMI Statements of Strategy, Annual Business Plans, Estimates and Annual Budget proposals, National

Continued relevance

Rationale

Impact Efficiency

Effectiveness

Inputs

Impacts

Results

Outputs

Activity

Strategic Objectives