a

- 1 -

Advanced PMM/RBB Training course

Council of EuropeMay 2005

a

- 2 -

Course objectives

1. Defining clear and informative performance indicators for preparing the 2006 PoAbudget & progress review/reporting

2. Review common difficulties with existing performance indicators

3. Introducing different types of indicators4. Ways towards using ‘common’ indicators?

a

- 3 -

Performance Indicators

What are they?� Essentially a ‘marker’;� Describing the results and effects of a

project or a programme in terms of quality and quantity over time.

� They are essentially indicators of project effects & success.

a

- 4 -

Performance indicators

Benefits to the project manager?� Agreed quality criteria for internal CoE services;

� Opportunity to build up ‘evidence’ on what does and doesn’t work – fosters organisational learning

� Enables manager to identify ‘good practices’ as well as ‘what the department is good at’;

� Provides a reference for discussions on the effectiveness of thedepartment;

� Helps defining the expected results.

a

- 5 -

Common problems in formulating performance indicators

a

- 6 -

Performance indicators too general

Often indicators are too vague, such as:

Participation of a significant number of parliamentarians and NGOs in the joint actions on Social Europe & North-South Dialogue

Data transmission method codified and transmission possible by the end of 2005 for most States which so wish

a

- 7 -

Additional information on quantities required

• There is no information on the initial percentage, therefore no comparison possible; check whether it is appropriate to indicate a date for the deadline.

• Often the quantities are just forgotten, e.g. number of hits on the Internet/Intranet portal.

• Don’t forget the quantification and the time frame if meaningful: 50% increase of hits … (compared to xxx hits in the year 2003) in the year 2004.

Be aware of the need for baseline data & for a clear definition of the scope of the intervention. Often one encounters indicators such as:80% of documents to be produced within the deadline

a

- 8 -

Quantitative but no qualitative info

Example: by 31 December 2005, the new database is operational

• There is no information which criteria this databank should comply with and for whom it is intended.

• No information on whether the introduction is successful (e.g. percentage of users being satisfied with the performance of the databank)

a

- 9 -

Performance Indicator = Activity

Is it possible to:

• specify the target group; • define the priority subjects, the general publications

and the main domains;• monitor whether the target group has received the

information.

General publications on priority subjects and the main domains of activity are made available to the citizen and target groups (expected result)A new graphic presentation is introduced; a new collection of booklets and leaflets is issued

a

- 10 -

How do you identify indicators?

Performance Indicators must correspond to certain criteria to ensure that:� Indicators are precise and relevant for the results

and the objectives of the project ( see logical framework);

� Provides information to key stakeholders;� Indicators determine what information needs to be

gathered and should, therefore, be selected on the basis of what data can be realistically and efficiently collected – or even collected at all.

a

- 11 -

Criteria 1: Relevant and Precise

• Relevant for the results and objectives they are describing in more detail;

• Precise by specifying what one intends to achieve and what that means in terms of the intended changes for specific organisations, groups of beneficiaries and situations.

Indicators are informing stakeholders about what the project intends to achieve and should therefore be:

a

- 12 -

Criteria 2: Achievable

Indicators should be based on a good knowledge of the actual situation and should therefore:Specify what is realistically achievable taking the context and the external factors into consideration (assumptions);

a

- 13 -

Criteria 3: Measurable and verifiable

• Specify the planned quantitative & qualitative changes over time with reference to the actual situation (base line);

• Ensure that such indicators are easily verifiable in practice by indicating how, where and by whomthey can be checked.

Indicators should be an easy way of checking if a project is achieving its aims and works according to plan:

a

- 14 -

Criteria 4: Selective and consistent

Monitoring can become a tedious and expensive task if not sufficient attention is paid to its manageability:

• Select only a limited number of key indicators for a number of key results;

• Whenever possible use the same key indicators to allow for a consistent, comparative analysis of results over time.

a

- 15 -

Using the LF to select indicators

Financial Resources

Performance

Indicator

Sources of

Verification

AssumptionsIntervention

Logic, PMM

Activities

Expected

Results

Project

Objective

Programme

Objective

Expected

Results

Activity Area

Objective

Strategic

Objective

Intervention

Logic, RBB

Activities

a

- 16 -

Expresses the long-term impact and the intended positive changes upon the beneficiaries (there can be multiple programme objectives).

Programme

Objective

Project

Objective

Defines the projects success and should therefore address the specific needs of the target group(s). Should be defined in terms of sustainable benefits or positive change for the target group(s).

Expected

Results

Describe the intended provision of services / products provided by the project to the target group(s).

Activities

Activities are the actions to be carried out in order to attain the Expected Results. They describe the budget in operational terms. Financial resources should be clearly attributed at this level by type of expenditure within the budgetary tables to be completed. The activities do not appear in the Logframe.

Objectives and results (PMM)

a

- 17 -

Expresses the long-term impact and positive changes upon the beneficiaries. (For Partial Agreements and administrative units with Votes I and VI, the Sub-Objective equates to the Strategic Objective)

Strategic

Objective

Activity Area

Objective

Defines the activities success. The Activity Area Objective should address the needs of the target group(s). It should be defined in terms of sustainable benefits or positive change for the target group(s).

Expected

Results

Describe the intended provision of services / products provided by the project to the target group(s).

Activities

Activities are the actions to be carried out in order to attain the Expected Results. They describe the budget in operational terms. Financial resources should be clearly attributed at this level by type of expenditure within the budgetary tables to be completed. The activities do not appear in the Logframe.

Objectives and results (RBB)

a

- 18 -

Progress chain and overlaps

Implementation of activities

Delivery of quality Services and Products

Use of Services and Products by intermediary

Satisfaction with Services and Products

Services reached all intended target groups / clients

Internal changes of the target group

Performance changes of target group

Use of services by final beneficiary

Satisfaction with services by final beneficiary

Change of behavior of final beneficiary

Benefits at general society

Results

Initial Impact

Long-termimpact

a

- 19 -

Progress organisational change

Training is taking place

Increased awareness of quality issues

Different ways of dealing with quality are compared

Some departments agreed to change and plan ahead

More departments have a clear action plan

Internal changes are being realised

Departments provide better quality services

Clients receive better products

Clients are more knowledgeable

Clients change their strategies

More citizens enjoy essential human rights

Results

Initial Impact

Long-termimpact

a

- 20 -

Types of indicators (1)

There are indicators that identify:

� Change/improvement processes�Socio-cultural, organisational, institutional�Understanding, insight, behaviour, relations

� Product development� Client benefit

� Technical, material

a

- 21 -

Identifying indicators (2)

Change (over time) can be indicated by:� Measuring and counting : numbers, %,ratios,.

� Comparison: quality benchmarking against an

accepted standard of service/product,

� Classification: yes / no of product/service availability,

suitability,....

� Qualitative descriptions of change processes

� A combination of the above.

a

- 22 -

Identifying indicators (3)

� You will usually need more than one indicator as it is rarely the case that one indicator can `prove’ anything.

� Trying to establish unequivocal causal relationships is extremely difficult and you usually need several different indicators before you can draw conclusions with confidence

a

- 23 -

Case Study Task:

1. Review the project objective and the results to be attained by the project by the end of next year (“ Progress Chain”)

2. Select and formulate indicators using the FOUR CRITERIA;

3. Present and discuss.