Lesson 2: Relevance and Coherence

Macerata, 13th October

Andrea Gramillano, t33 srl

Agenda

❖What we have to keep in mind from the previous lesson?

❖What does relevance mean?

❖The SWOT and PEST analysis

❖Coherence

What do we need to keep in mind?

The Project / Program cycle

Identification

Formulation

Implementation

Evaluation and review

The Chain of Public Interventions

Policy

Program

Project (1) Project (2)

Other delivery policy tools: law,

tax, agency, campaign.

Some references for defining the theoretical concepts (1)• Policy: A deliberate plan of action to guide decisions and achieve rational

outcome(s). The term may apply to government, private sectororganizations and groups, and individuals. Presidential executive orders,corporate privacy policies, and parliamentary rules of order are allexamples of policy. Policy differs from rules or law. While law can compel orprohibit behaviours (e.g. a law requiring the payment of taxes on income)policy merely guides actions toward those that are most likely to achieve adesired outcome.

• Programme (policy delivery tool): the donor defines criteria, priorities,objectives, call for projects and allocate a budget for a set of projects,without saying which projects

• Projects: have sharp costs, clearly identified beneficiaries, timing andactivities, output and outcomes

Some references for defining the theoretical concepts (2)• The identification phase delivers a project idea (a pre-feasibility study)

and identifies all the problems and possible solutions.

• The formulation phase is after the identification phase: projectapplicants’ delivery is a sort of feasibility study, explaining why andhow the proposed project is well-grounded and is likely to reach theobjectives and results and in particular to address the challenges andneeds of beneficiaries and target groups. The main purposes are:confirm the relevance and feasibility of the project idea (seeidentification phase); provide more details about technical,operational, economic, financial aspects…; prepare a financialproposal and a financing decision.

Between the identification /formulation and implementation phase there is the financingdecision• Before making a financing decision, the donor usually conducts an ex-

ante assessment of the relevance and coherence of the project. The financing decision defines the end of the formulation phase and start-up of the implementation.

Relevance and Coherence ( EX ANTE)Needs

Resources (inputs)

Output (implementation)

Result

(Specific Objective)

Programme /

policy

Relevance

External coherence

Internal Coherence

Utility

Utility / Relevance: Definition

1. The term RELEVANCE, in the context of an evaluation, refers to theappropriateness of the explicit objectives of the program in relationto the socio-economic problems it is supposed to address.• Is the project justified in relation to the needs?

• Can their raison d'être of the project still be proved?

• Are the expected or unexpected effects globally satisfactory from the point ofview of direct or indirect beneficiaries?

2. UTILITY : Have the expected effects been produced?

3. Sources of information: interviews, context indicators, documentsand studies

Analytical tools for relevance

• SWOT and PEST

Analytical tools: PEST

• PEST analysis is a tool from business science. It is a useful tool for understanding the general feature of the Context.

• PEST is an acronym for Political, Economic, Social and Technological factors which provide a framework for reviewing a situation, or a strategy.

• It is very simple, and is a good subject for work group session or brainstorming meeting.

• PEST factors are essentially external, completing a PEST analysis is helpful prior to completing a SWOT analysis

PEST MATRIX CRITERIApolitical•ecological/environmental issues•current legislation home market•future legislation•international legislation•regulatory bodies and processes•government policies•government term and change•trading policies•funding, grants and initiatives•home market lobbying/pressure groups•international pressure groups•wars and conflicts

economic•home economy situation•home economy trends•overseas economies and trends•general taxation issues•taxation specific to product/services•seasonality/weather issues•market and trade cycles•specific industry factors•market routes and distribution trends•customer/end-user drivers•interest and exchange rates•international trade/monetary issues

social•lifestyle trends•demographics•consumer attitudes and opinions•media views•law changes affecting social factors•brand, company, technology image•consumer buying patterns•fashion and role models•major events and influences•buying access and trends•ethnic/religious factors•advertising and publicity•ethical issues

technological•competing technology development•research funding•associated/dependent technologies•replacement technology/solutions•maturity of technology•manufacturing maturity and capacity•information and communications•consumer buying mechanisms/technology•technology legislation•innovation potential•technology access, licencing, patents•intellectual property issues•global communications

Analytical tools: SWOT

• SWOT is an acronym for Strengths, Weaknesses, Opportunities, Threats.

• A SWOT analysis is a subjective assessment of data which is organized by the SWOT format into a logical order that helps understanding, presentation, discussion and decision-making. The four dimensions are a useful extension of a basic two heading list of pro's and con‘

• Strengths and Weaknesses are internal factors , whereas

• Opportunities and Threats might be: • Obstacles or Advantages for strengths and weaknesses coming from external

factors• The evolution in the near future

SWOT Analysis

NOW Strengths Weaknesses

IN THE

FUTURE

Opportunities

(external driving

forces empowering

the strengths )

Threats

(external obstacles)

PEST AND SWOT COMBINATION

• PEST helps the identification of SWOT factors.

• PEST assesses more generally the global situation.

• SWOT is an assessment of the more specific sector / territory or policy / project / programme.

Practical example (1) Assessment the relevance of the issues: PEST

PEST

Analysis

factors

Importance / Impact/ Time frame

Political

Economical

Social

Technological

Relative Importance:

- Critical

- Important

- Un-important

- Unknown

Impact

• Very positive

• Positive

• Medium

• Negative

• Very negative

Time frame (short /

long term)

SWOT

P

E

S

T

StrengthsHere you have to locate the items that in the PEST Are short term important / criticalPositive/ very

positive

WeaknessesHere you have to locate the items that in the PEST Are short term important /

criticalNegative / Very

negative

OpportunitiesHere you have to locate the items that in the PEST Are long term important / criticalPositive/ very

positive

ThreatsHere you have to locate the items that in the PEST Are Long term important / criticalNegative / Very

negative

POLITICAL

ECONOMIC

SOCIAL

TECHNOLOGICAL

FROM SWOT TO NEEDS

S W

T OVantaggi comparati

COMPARATIVE ADVANTAGE

STRUCTURAL GAPS

Verify the consistency of the analysis of needs

S T

W O

• Obstacles (needs)

• Potential for solutions

Coherence: Definition

• Coherence has two dimensions:

• Internal: correspondence between the different objectives of the same intervention. Internal coherence implies that there is a hierarchy of objectives, with those at the bottom logically contributing towards those above.

• External: correspondence between the objectives of an intervention and those of other public interventions which interact with it.

Example: if a national policy and EU socio-economic programme are implemented in a complementary manner in the same territory for the purpose of developing SMEs, it can be said that there is external coherence. (European Commission – DG REGIO EVALSED GUIDE)

INTERNAL COHERENCE

The internal coherence shall investigate:

• LOGIC of the intervention ( link betweenobjectives – activities - inputs)

• SINERGY among different activities of the project or CONFLICT

• FINANCIAL allocation

STRATEGY

Activity A

Resources

Activity B

EXTERNAL COHERENCE: TWO DIMENSIONS

VERTICAL Coherence

It is the consistency of the project withthe above superordinated funding Program and (National – International) Policy

HORIZONTAL Coherence

It is the relation with other parallelinterventions or mainstream policies(Environment – Gender Issue)

The External Coherence Evaluation Process

STEP 1

Define the subject of investigation

STEP 2

Identification of relevant documents

STEP 3

Conceptual Analysis

STEP 4

Level of Coherenceassessment

Step 5

Final Assessment

STEP 1Define the subject of investigation

Vertical coherence between the project and:

• the funding programme

• the related policy

Horizontal coherence:

• at the local level, with other sectors or project

• at the regional level, with other Plans or Programmes

• at the national level across ministries,

• at the international with EU or other donors

STEP 2Identify Key documents

• Vertical coherence: identification of strategic Programme and Policy documents which describe the organisation, goals, tools or instruments that are being used.

• Horizontal coherence: identification of other Policy documents, which spell out the main actors or organisations identified to play key roles in implementing the objectives of the organisation.

• The common features of each of the policy documents can be categorised in terms of Strategy (overall themes WHY? IN WHICH FIELDS?), Organisation (institutions responsible for achieving integration WHO IS DOING WHAT?) and Instruments (e.g. - ways to achieve these goals HOW?).

STEP 3Conceptual Analysis

Once these documents have been identified, key words or concepts regarding 1) the strategy, 2) the organisation, and 3) the instruments

There will therefore be key words associated with the Strategy, Organisation and Instrument.

Once key words have been selected for the document, the next step involves checking Evaluated Project for the key words of other related strategic document.

Example: the Key words of the documents of The Funding Program are: Innovation – R&D –SME support. The key word have to be checked in the project document referring to 3 main categories: Strategy, Organisation and Instruments

STEP 4Screening the level of coherence

Each time the key words occur in the checked document, the relation between the project and the programme has to be verified .

The screening has to be done for both: vertical and horizontal dimension.

The aspect to take first in consideration is Strategy for which the elements to use for the screening are slightly different (see the following slide).

The aim is to see if the key word : • shows risks of conflict or • overlapping • Defines different areas of actions – acknowledgement• Implies some form of coordination, complementarity• Assures synergies - integration

EXTERNAL COHERENCE: Strategy

Horizontal (with other interventions)

Vertical (with the Program/ Policy Funding)

CONFLICT The Project Objectives could negatively impact on the Strategy

The Project Objectives could negatively impact the Strategy

RISK OF OVERLAPPING / INCONSISTENCY

The Project Objectives are in the same policy domain but without relation or awareness

The Project Objectives are inconsistent with the Strategy

ACKNOWLEDGMENT The Project Objectives take in consideration explicitly the other interventions

The Project Objectives refers the “superior” Strategy

COMPLEMENTARITY The Project Objectives identify a specific area of interaction

The Project Objectives represent an execution of the superior strategy on the specific sector/ territory

INTEGRATION The Project is explicity a “follow up” , an “execution”, or “part of” other interventions

The Project Objectives are functionallyrelated to the superior Strategy and represent a “condition sine qua non” for its realization

EXTERNAL COHERENCE:Organisation and Instruments

ORGANISATION INSTRUMENTS

CONFLICT The Project organisation does not recognise the other Institution or is not eligible by the Funding Program (e.g. conflict of interest)

The Instruments used by the project are not compatible (not eligible)

RISK OF OVERLAPPING/ INCONSISTENCY

The Project Organisation works in the same area without taking in consideration the administrative – institutional linkages

The Instruments adopted are not aligned (too old – too expensive - etc.)

ACKNOWLEDGMENT The Project Organisation identifies ways of communication and exchange of information “unatantum”

The Instruments are aligned and compatible

COMPLEMENTARITY The Project Organisation establishes stable form of cooperation (e.g. joint meeting)

The Instruments can work in parallel in a complementary and synergistic manner

INTEGRATION The Project Organisation is:- part of the same Administrative body - exchange of same human resources -Adopt a common management system

The Instruments are interconnected by financial and/or administrative point of view

STEP 5FINAL ASSESSMENT

The final step consists in weighting the level of coherence for the three aspects

Strategy: Are there any similarity on the strategic level? (=same/similar/ objectives)

Organization: Are the actors involved in the implementation the same/similar?

Instruments: • Conflict = - 3

• Risk of Overlapping = -1

• Acknowledgement = 1

• Coordination = 2

• Complementarity = 3

Example

VERTICAL HORIZONTAL

Programme Policy Regional Plan Project “A”

S O I S O I S O I S O I

ProjectUnder Evaluation

3 3 2 2 2 0 1 1 2 3 2 1

STEP 5FINAL ASSESSMENT

SUMMATIVEThe scoring can provide the following

judgement for each dimensions (external –Internal):

A) The overall level of Coherence (total average of the scoring), e.g. whether the project has “full complementarity” or, on the opposite side, is “rather conflictual”

B) The level of coherence for different aspects (Strategy – Organization –Instruments)

C) The level of coherence for different interventions (Policy –Program – other interventions)

FORMATIVE• The scoring can identify where there is room

for improvement in term of better cooperation or exploit potential synergies.

For example: there is a potential for collaboration between the evaluated project with a regional plan or Project

WORKOUT n.1 (Relevance)

Population for the SWOT

• Demographic downward trend during the 1980s and 1990s with slight recent recovery

• Continuous increase in region’s working age population

• Projections indicate that the North East total population will continue to increase over the next two decades due to internal migration

• Population profile is ageing.

• Working age population decline has implications for future labour supply in the North East, as well as for the size and segmentation of markets within the region.

Economy for the SWOT

• Although there has been some growth in the knowledge economy, many new service sector jobs are in low value-adding private sector activities

• The experience of the early 1990s suggests that the overall pattern of recent employment growth is unlikely to be resilient in the event of recession.

• The shrinkage of traditional manufacturing in the North East emphasises the need to develop higher value-added products and services using the region’s science and technology strengths.

• Long-standing enterprise deficit.

• Qualifications profile has been improving in recent years, but skill levels remain low in qualifications.

Nature and environment for the SWOT

• Good quality of life and presence of natural amenities such as national parks to attract people to live, work and invest in the region.

• Challenge of future economic development is to create wealth while reducing environmental impacts.

• Substantial capacity to develop the renewable energy sector, which represents both an economic opportunity (business growth/start-up potential) and environmental opportunity (clean and lower carbon energy).

• National programmes have planned to invest additional resources to promote renewable energy production

• Private investors will invest in the field of coal and oil extraction and exploitation

MAKE the SWOT Analysis

NOW Strengths Weaknesses

IN THE

FUTURE

Opportunities

(external driving

forces empowering

the strengths )

Threats

(external obstacles)

WORKOUT n.2 (Coherence)

EXTERNAL COHERENCE – Focus: strategyand objectiveIs the project coherent with…?Coherence

Intensity →Strategy (objective)

Organisation (Activities)

Instruments (Type of support)

ProgrammeStrategy

Prog. I priority

Other policy (RES)

Other projects (horizontalcoherence)

Programme «International Development Programme»

The International Development Programme for 2007-2013 provides an excellent opportunity for the region “POOR” to further progress its economic development, through priorities determined by the region itself and defined by a strong regional partnership drawn from all parts of the region. In addition to International Development Programme, the Regional Economic Strategy (RES) represents the “POOR’S” response with the following targets:

• To raise the region’s GVA per head from 80% to 90% of the national average

• To get between 61,000 and 73,000 more people in the region into work

• To create between 18,500 and 22,000 new businesses

Programme Priorities

The International Development Programme comprise three priorities:

• Priority One: Enhancing and Exploiting Innovation (53%).

• Priority Two: Business Growth and Enterprise (43%)

• Priority three: Addressing Climate Change

Accordingly, through Priority 1, the strategy will:

• Increase the region's value-added by investing in ‘opportunity’ to exploit the region's science and technology strengths to take advantage of growing marketsI

• Invest in integrated projects that, on one level, will strengthen capacity to apply science, technology and innovation in businesses.

The programme will invest exclusively by using financial instruments as:

• Guarantee schemes in combination with grants

• Loans

• Venture Capital

Project A

Objective Improve the capacity of social enterprisesto develop and deliver sustainable andhigh impact innovations

Activities Research on effective business models

Pilot test on new business models,evaluation of business models

Establishing network

Building a motorway

Budget Estimated budget 3,800,000.00 Euro

Type of support

Guarantee scheme without anycombination with grant

Objective Introduce technologies, methodsand tools to organise this circularapproach to sediment management.

Activities Organisation of meetings anddrafting of manual for circulareconomy (e.g. recycling)

Pilot tests

New investments

Budget Estimated budget 5,800,000.00 Euro

Type of support

Grant

Project B

External coherence: - Is the project A (B) coherent with the programme? (vertical coherence)- Is the project A coherent with project B? (horizontal coherence)

- In terms of objective, activities, and type of support?

Internal coherence- Has the project A (B) internal coherence?

- Objective and activities?- Activities and budget? the programme? (vertical coherence)

- Is the project A coherent with project B? (horizontal coherence)

See you

www.t33.it

a.gramillano@t33.it