MetronomeMetronome

A METhodology foR evaluatiON of prOject iMpacts in the fiEld of Transport

Grant No. 213546

D4.1 Sustainable Development methodology development and applicationresults

Project: METRONOME 213546Document Number and Title: D4.1Work-Package: WP 4Deliverable Type: ReportDate of Delivery: June 2009Author/Responsible(s): María López-Lambas, Elena López-Suárez,

Lissy La Paix Puello.Contributors: Anne Binsted, Anu Tuominen, Tuuli JärviApproval of this report: Anu Tuominen, VTTKeyword List: project evaluation, sustainable develop-

ment, research project impacts, methodol-ogy

Dissemination level: Public (PU)

2/61

2

Change History

Version Date Status Author (Partner) Description1 .2009 draft UPM, TRL2 23.7.2009 draft UPM, TRL, VTT3 30.7.2009 final UPM, TRL, VTT

Distribution list

European Commission: frank.smit@ec.europa.eu

METRONOME partners:anu.tuominen@vtt.fituuli.jarvi@vtt.fijan.vander.waard@rws.nlemit@certh.granatolij.sitov@cdv.czabinsted@trl.co.ukamonzon@caminos.upm.esmelopezlambas@caminos.upm.es

METRONOME STEERING GROUP:

heikki.kanner@vtt.fifloortje.hanneman@rws.nlgea@certh.grjosef.mikulik@cdv.czhdalkmann@trl.co.ukmromana@caminos.upm.es

3/61

3

Table of contents

1 INTRODUCTION .......................................................................................................................... 4

1.1 OBJECTIVES AND SCOPE .............................................................................................................. 41.2 BACKGROUND ............................................................................................................................. 4

2 PREVIOUS WORK ON EVALUATING CONTRIBUTION TO SUSTAINABLEDEVELOPMENT..................................................................................................................................... 5

2.1 AIM, SCOPE AND APPROACH ......................................................................................................... 52.2 THE SCOPE OF SUSTAINABLE DEVELOPMENT .................................................................................. 62.3 RESEARCH PROJECT EVALUATION METHODOLOGIES ....................................................................... 102.4 CONCLUSIONS .......................................................................................................................... 12

3 SUSTAINABLE DEVELOPMENT EVALUATION METHODOLOGY ..................................... 13

3.1 EVALUATION PROCESS ............................................................................................................... 133.2 SELECTION OF FP5 AND FP6 PROJECTS AND DATA COLLECTION ..................................................... 143.3 EVALUATION MATRIX: PROJECT RESULTS VS. SUD OBJECTIVES ...................................................... 15

3.3.1 Identification of research objectives ............................................................................... 153.3.2 Impact indicators.............................................................................................................. 163.3.3 Evaluation matrix.............................................................................................................. 17

3.4 EVALUATION OF TRANSFERABILITY OF KNOWLEDGE ....................................................................... 193.4.1 Impact indicators.............................................................................................................. 193.4.2 Data for impact evaluation: the questionnaire ............................................................... 193.4.3 Data analysis..................................................................................................................... 22

4 METHODOLOGY IMPLEMENTATION .................................................................................... 23

4.1 PROJECT SCREENING ................................................................................................................. 234.2 DATA COLLECTION .................................................................................................................... 254.3 RESULTS FROM THE PROJECT EVALUATION MATRIX ....................................................................... 254.4 QUESTIONNAIRE SURVEY RESULTS .............................................................................................. 29

4.4.1 Sample Characteristics ..................................................................................................... 294.4.2 Results............................................................................................................................... 30

4.5 INTEGRATION OF ASSESSMENT RESULTS ...................................................................................... 344.5.1 Integrated Analysis by Financial Instrument .................................................................. 37

DISCUSSION....................................................................................................................................... 38

4.6 METHODOLOGICAL DEVELOPMENT ............................................................................................... 384.7 RESULTS .................................................................................................................................. 40

4.7.1 Project performance......................................................................................................... 404.7.2 Knowledge transferability ................................................................................................ 40

5 CONCLUSIONS AND RECOMMENDATIONS ........................................................................ 42

5.1 RESULTS .................................................................................................................................. 425.2 CONCLUSIONS .......................................................................................................................... 43

5.2.1 Results............................................................................................................................... 435.2.2 Performance Score ........................................................................................................... 445.2.3 Transferability score......................................................................................................... 445.2.4 Financial instrument ......................................................................................................... 45

5.3 EXTENSIONS AND IMPROVEMENTS OF THE PROPOSED METHODOLOGY ............................................. 45

ANNEX I ............................................................................................................................................... 49

ANNEX II ............................................................................................................................................. 56

4/61

4

1 INTRODUCTION

1.1 Objectives and Scope

The objective of METRONOME Work Package (WP) 4 is the development of a Methodol-ogy, which will be applied in the evaluation and assessment of Framework Programmes5 and 6 transport projects contributing to Sustainable Development (SuD) and address-ing societal problems. The ultimate aim of the evaluation results and findings from FP5and FP6 is to help in definition of new research policy objectives and intermediate per-formance targets for FP7 and further, generate new knowledge relevant to societal, envi-ronmental and economic transport related issues.

To do so, a methodological framework was developed and the methodology was appliedto a sample of projects. This Deliverable D4.1 is the main output of the work carried outin METRONOME WP 4.

1.2 BackgroundIt is only in recent years that there has been a general shift in the focus of research froman emphasis on scientific quality and technological objectives towards strategic aspectsand the relevance of research from a practical perspective (Luukkonen, 1997). Theevaluation of research in terms of its socio-economic impacts, particularly from a quanti-tative point of view, is also a relatively recent shift (Capron et al, 1997).

The concept of sustainable development is itself relatively recent having been introducedin 1987 by the Brundtland Commission in their report ‘Our Common Future.’ It is alsoonly in recent years that there has been a recognised need to evaluate research projectsin terms of their contribution to sustainable development.

Sustainable development is a difficult concept to evaluate empirically as it is a multi-faceted concept that has been interpreted in a number of ways (Widmer, 2002). Thechallenge of evaluating the contribution of research projects to sustainable developmentis complicated further by the fact that projects are often designed to contribute to cer-tain aspects of sustainable development rather than the broad concept as a whole. Thishas led to the development of a relatively wide range of social, economic, quality of lifeand environmental criteria against which to evaluate the impact of economic, social andnatural science research projects on sustainable development.

5/61

5

2 PREVIOUS WORK ON EVALUATING CONTRIBUTION TOSUSTAINABLE DEVELOPMENT

2.1 Aim, scope and approachThere are a large number of comprehensive indicator sets that can be used to evaluatethe impact of transport scheme and policy proposals upon sustainable development, butthere are an extremely limited number of ‘sustainable development’ evaluation criteriathat are suitable to evaluate research projects against. In Europe there is now a com-prehensive suite of legislation and standards in place to ensure that the impact of pro-jects, and their alternatives, upon sustainable development is fully considered before im-plementation, but there is no such standardised approach to evaluating the impact ofresearch on sustainable development. This could be considered surprising given the in-creasing emphasis on the accountability, effectiveness and efficiency of Governmentfunded research, and on sustainable development.

Widmer (2002) states that the following three factors contribute to the lack of indicatorsets to measure the contribution of research projects to ‘sustainable development’against:

The long-term nature of the sustainable development concept;Uncertainty being a key component of the concept; andThe holistic nature of the concept makes it difficult to break down without being re-strictive.

Evaluating the contribution of research projects to sustainable development is challeng-ing, but there is a need for a standardised set of evaluation criteria to enable this to bedone efficiently, consistently, objectively and transparently on a European level. Thistask has therefore searched extensively to identify indicators that are used to evaluatethe impact of research projects upon sustainable development.

This section will review indicator sets that are used to evaluate ‘sustainable develop-ment.’ In recognition of the lack of publicly available indicator sets against which toevaluate the contribution of research projects to sustainable development we have in-cluded a number of indicator sets with broader remits. These will help to identify thebroad range of impacts that should be considered on a European level thereby helping toensure that the methodology developed in WP4 considers all of the necessary facets ofsustainable development.

A wide range of sources were searched in order to try and identify evaluation method-ologies used to help determine the impact of research projects upon sustainable devel-opment. Our search was limited to indicator sets published in the English language.

The following databases were referenced for our review:TRL Library KnowledgeBase (English language International Transport ResearchDocumentation (ITRD)1 + Library catalogue2);

1 ITRD is a multi-lingual biblioGraph database that contains citations to international literature, research pro-jects and computer programmes on all aspects of road research, transport, and traffic planning. The data-base contains approximately 850 journals from 40 countries as well as books, reports, dissertations, patents,conference proceedings, and standards and specifications.2 TRL’s Library & Information Centre has collections of published materials in the form of books, periodicals,conference proceedings, standards, statistics, guidance notes and several thousand TRL Research Reports.

6/61

6

ScienceDirect;Transportation Research Information Services (TRIS).3

Extensive online searches were also conducted using internet search engines. Referencelists in relevant documents were also reviewed in the hope of identifying other indicatorsets.

The searches were limited to evaluation methodologies used in Europe. There are nu-merous methodologies that have been developed to evaluate contributions to sustainabledevelopment in the developing world, but these were not considered to be transferableto the review of FP5 and FP6 transport research projects or the European context.

Evaluation methodologies that focus upon one aspect of sustainable development ratherthan the concept as a whole have also been excluded. This is because the very natureof sustainable development means that evaluations of impact upon sustainable develop-ment require an integrated approach. Indicator sets focusing solely on one specific as-pect of sustainable development, such as environmental impact, also contain far moredetail than can be accommodated in an overarching evaluation framework. This has re-sulted in the necessary exclusion of many of the indicator sets identified.

The lack of methodologies for evaluating the contribution of research projects to sustain-able development has led to the inclusion of a number of indicator sets that have notbeen devised to review research projects against, but that need to be taken into accountfor the development of a methodology to review FP5 and FP6 projects against. Theseare outlined within section three of this report.

2.2 The scope of Sustainable DevelopmentA standardised set of criteria used to evaluate the contribution of an FP5 or FP6 researchproject to sustainable development must take into account the full range of potentialsustainability impacts and be based upon up-to-date knowledge of sustainability (Langeret al, 2003). To determine whether it has been value for money, it must also reflect EUobjectives and guidelines. This section therefore summarises key criteria that a Euro-pean methodology should enable the contribution of research projects to be evaluatedagainst.

The documents reviewed below, which contain guidelines and externally developed sci-entific and standardised criteria defining the scope of sustainable development, consti-tute the external evaluation framework in which FP5 and FP6 research projects exist.The criteria outlined below should therefore be used as the basis for a coherent set ofevaluation criteria and indicators.

Agenda 21 (UN, 2005)Agenda 21, a programme operated by the UN, is a comprehensive plan of actions thatsets out requirements for action on local, regional, national and global scales within allfields that have an impact on the environment. The Agenda, which is based upon the

3 TRIS is the largest and most comprehensive biblioGraph resource on all modes and disciplines of transpor-tation. The database is produced and maintained by the Transportation Research Board (TRB) and containsover 640,000 records of published research.

7/61

7

concept of sustainable development, was adopted by more than 178 Governments at theUnited Nations Conference on Environment and Development (UNCED) in 1992.

The Agenda is comprised of four sections, the last of which called for the development of‘sustainable development’ indicators against which to evaluate the effectiveness of pro-cedures and practices in terms of meeting sustainable development objectives. The 21indicators that were developed are based upon analysis at the local level, although theyembody the key requirements of sustainable development on a broader level.

When evaluating the contribution of European research projects to sustainable develop-ment, the following criteria should therefore be incorporated:

Protection of the countryside and building up resources;Social equality;Sustainable-development pattern;Sustainable mobility;Policy measures to reduce emissions of greenhouse gases;Prevention of toxic emission harmful to the environment;Local autonomy;Popular participation.

EU Renewed Sustainable Development Strategy (EC, 2006)The EU adopted a revised sustainable development strategy in June 2006. The strategyviews sustainable development from an operational perspective and focuses policy objec-tives around seven key challenges, which are identified as priority areas to be addressed.An evaluation methodology should recognise these seven key challenge areas and de-termine whether FP5 and FP6 research projects have made a positive contribution to-wards them. The seven areas are as follows:

Climate change and clean energy;Sustainable transport;Sustainable consumption and production;Conservation and management of natural resources;Public health;Social inclusion, demography and migration;Global poverty and sustainable development challenges.

A set of evaluation criteria should also, as a minimum, incorporate the four key objec-tives outlined in the strategy. These are:

Environmental protection;Social equity and cohesion;Economic prosperity; andMeeting international responsibilities.

The Strategy states that a process orientated set of criteria, as advocated by Langer at al(2002), should focus upon the ten policy principles guiding these objectives, namely:

Promotion and protection of fundamental rights;Solidarity within and between generations;Open and democratic society;Involvement of citizens;Involvement of business and social partners;Policy coherence and governance;Policy integration;

8/61

8

Use best available knowledge;Precautionary principle (taking action now to avoid potential high level damage inthe future);Make polluters pay.

EU SDI indicator set (EC, 2005)The EU SDI indicator set was adopted in 2005 (EC, 2005) and renewed in 2007 (Euro-stat, 2007) to reflect the renewed EU Sustainable Development Strategy (EC, 2006).Developed by the Sustainable Development Indicators Task Force it encapsulates themost scientific knowledge and understanding of sustainable development along with theEU’s priority sustainable development policy issues. It also incorporates other key indica-tor initiatives for completeness and to harmonise approaches. The indicator set forms aframework within which progress towards goals and objectives can be determined. It isin recognition of the fact that project objectives will vary that the list of criteria is broadto enable them to be adapted.

The sustainable development indicator set encompasses ten themes, each of which con-tains a range of socio-economic and environmental issues and correlates with one of theseven challenges detailed in the EU Sustainable Development Strategy (EC, 2006). Thethemes, which should be reflected in a methodology to evaluate the contribution ofEuropean research projects to sustainable development, are:

Socio-economic development;Sustainable consumption and production;Social inclusion;Demography changes;Public health;Climate change and energy;Sustainable transport;Natural resources;Global partnership; andGood governance.

The EU SDI is comprised of a large number of indicators in recognition of the multidi-mensional nature of sustainable development. These are sub-divided into three sets.The 12 high level indicators, which correspond with the EU’s ‘overall’ sustainable devel-opment objectives as outlined in core EU policy documents, will be of most relevance tothe development of the WP4 methodology. They provide a robust overview of impactupon key challenges that the EU sustainable development strategy is seeking to address.Along with the other indicators developed, these are continually developed and refined.

Sustainable development indicators used by national and internationalagencies (OECD, 2002)There are many inconsistencies evident in the national sets of indicators used in Euro-pean Member State Sustainable Development Strategies (Eurostat, 2007). This stemsfrom the fact that national indicators were developed before the European wide indicatorset was published, and so it is likely that greater consistency can be expected in the fu-ture.

In 2002 the OECD compared the sustainable development indicators used by a numberof countries to review progress towards sustainable development on a national level.They based their analysis on UNCSD categories and themes, which they deemed to bekey to the evaluation of sustainable development. These themes, which provide another

9/61

9

perspective from which FP5 and FP6 transport research projects could be evaluated, areoutlined below:

Social:Equity;Health;Education;Housing;Security;Population.

Environmental:Atmosphere;Land;Oceans, seas and coasts;Fresh-water;Biodiversity.

Economic:Economic structure;Consumption and production patterns (i.e. energy use).

Institutional:Institutional framework (i.e. strategic implementation of sustainable develop-ment);Institutional capacity (i.e. access to information).

The DAC Guidelines: Strategies for Sustainable Development (OECD, 2001)These ‘good practice’ guidelines were formulated by the OECD DAC Working Party onDevelopment Co-operation and Environment, a task force chaired by the UK Departmentfor International Development (DfID) and the European Commission. They were pro-duced in recognition of the fact that ten years on from the UNCED, very little guidanceexisted to help Governments fulfil their commitments. This guidance has an internationalfocus and is based upon international experience (of both developed and developingcounties) and multi-stakeholder reviews. It is nevertheless relevant to the Europeancontext as it contains a set of principles that encompass desirable processes and out-comes for ensuring sustainable development.

The guidance does not contain a set of indicators against which the contribution ofstrategies towards sustainable development can be evaluated. The guidance states thatthis is because no two strategies will ever be the same, and that different countries willhave different views about what issues and actions should be considered when evaluat-ing impacts to sustainable development. It does, however, indicate that a focus uponthe following considerations could provide the most effective basis for methodology crite-ria:

Consideration of the key component processes and mechanisms (within the rele-vant country/ies) which are considered to be relevant to sustainable development(i.e. awareness, communication, participation in sustainable development actionand debate, capacity-building, and co-ordination between sectors);

10/61

10

Analysis of the quality of these processes and mechanisms in terms of their con-tribution to effective change towards sustainable development (i.e. change in at-titudes, influence on decision making, and societal awareness);Progress towards key sustainable development indicators (i.e. impact of the pro-ject);Suggestions for improving and strengthening planning processes for sustainabledevelopment.

2.3 Research project evaluation methodologiesThis section looks at different approaches that have been undertaken for evaluating theimpact of research projects on sustainable development.

Implementing Evaluation in the context of Sustainable Development: ThePlanning and Commissioning Procedure of Evaluations with SustainableDevelopment as part of a Tool Box (Langer et al, 2003)Researchers at the Vienna University of Economics and Business Administration haveproduced this ‘Tool Box’ (Langer et al, 2003), which was funded by the Austrian ScienceFund. In recognition of the fact that there are relatively few methodologies that evalu-ate the contribution of research projects to sustainable development, the Tool Box drawson general evaluation theory and practice from other fields of evaluation to suggest howcontributions to sustainable development can be evaluated.

The Tool Box does not provide detailed criteria that methodologies should be based uponas it highlights the importance of the criteria reflecting not only externally developedstandardised criteria but also case specific criteria (i.e. the internal objectives of FP5 andFP6 programmes). It also highlights the role of the client in setting context specific crite-ria. The Tool Box does, however, provide guidance on the type of criteria that should beused. These are:

What is the contribution of the program towards sustainable development?How does the program in question contribute to sustainable development withina defined area?What progress was achieved relative to defined objectives of sustainable devel-opment?Are the structures in the program adequate in order to contribute towards sus-tainable development?Are the structures in the program suitable to achieve the desired contribution to-wards sustainable development?Is the design, e.g. of a process, suitable to contribute towards sustainable devel-opment?Is the design, e.g. of a process, suitable to achieve the desired contribution tosustainable development?

Some of these questions are broad, focusing on the objective of sustainable develop-ment as a whole, whilst others relate to specific (sustainable development related) objec-tives that the research being evaluated was commissioned to address.

The Framework of Sustainable Development (Langer et al, 2002)The Framework was designed as a ‘comprehensive system of reference’ to enable pro-jects to be evaluated in terms of the extent to which they consider sustainable develop-

11/61

11

ment processes. The Framework, which covers nine different aspects of sustainable de-velopment, was formulated in recognition of the fact that most other sustainable devel-opment indicator sets only give an insight into the impacts of a project, not the relatedprocesses that are integral to the sustainable development concept. The criteria onwhich the Framework is based are flexible in recognition of the need to accommodatethe demands of evaluators and stakeholders.

In the Framework, each of these aspects of sustainable development is accompanied bythree statements that, when used for evaluation purposes, enable evaluators to demon-strate the extent to which each of the aspects has been considered in the research.

The criteria outlined in the Framework are as follows:

Integrated aspects (does the project take a single or multi-faceted interrelatedapproach to different aspects of sustainable development);Temporal aspects (should consider different time-spans, duration of effects of theproject, consider amount of time systems take to react, intergenerational equity);Spatial aspects (boundaries, interrelations with external systems, spatial interac-tions, scales from the local to global);Dynamic change and risk (uncertainty, trade-offs, decision-making, anticipatingchanges, managing impacts);Levels and limits (of natural, social and economic resources and capacities, sub-stitutability);Distribution (in terms of equity, social systems, opportunities, disbenefits);Capacity building through co-operation (stakeholder participation, knowledgenetworks, networking, spatial and temporal levels);Participation and governance (policy, levels of governance, information, empow-erment); andReflexivity (system conditions, controlling, monitoring, adapting, learning).

Effectiveness of national transport strategies as a means to promote thedevelopment of more sustainable transport systems (Transport Mistra,2008)The IMPACT (Implementation Paths for ACTion towards sustainable mobility) project,which is being funded by the Swedish Transport Mistra programme, was commissionedto develop models and tools for the support of sustainability oriented decision-makingand implementation in the transport sector.

In recognition of the fact that sustainability cannot be limited to one specific scientificdiscipline IMPACT is comprised of a number of different Common Focal Points (CFPs)that encompass the various disciplines relevant to sustainability in the transport sector.The results of the project will enable the evaluation of how specific strategies can be im-proved to contribute more to sustainable mobility.

The project was due for completion in 2008 and the project outputs (tools and models tosupport sustainability oriented decision making) have not yet been made publicly avail-able. A recently published paper (Kimble and Tight, 2007) does, however, report on theprogress of the IMPACT project. One of the focus’ of this paper is how high level plansand research programmes contribute to the development of ‘more sustainable transportsystems.’ The research details that two criteria need to be fulfilled if a programme canbe said to have contributed to sustainability;

12/61

12

‘Contribute overall to reduce pressures on natural life support systems (includingthe climate system), that are overexploited with regard to their continued supportof society’s needs, or are at the risk of becoming overexploited within the effec-tive lifetime of the change or policy’ and;‘Contribute overall to increase the well-being in a broad sense of the presentgenerations, avoiding entirely negative effects on people living in absolute pov-erty.’

The paper also states that to contribute towards sustainable mobility, projects should be‘designed and implemented’ with the participation of stakeholder groups.

2.4 ConclusionsMarkus et al. (2003) caution that there is currently insufficient knowledge to enable thedevelopment of evaluation criteria that are conducive to making valid statements aboutthe contribution of research to all aspects of sustainable development. The authors havetherefore called for further research to fill the gaps in knowledge that currently restrictthe ability to comprehensively evaluate the impacts of research upon sustainable devel-opment. They state that ‘methodological difficulties in terms of measurement, valuationand comparability’ are restricting progress in this field, which is currently supported by‘little practical experience.’ This is a contention backed up by the UN Division for Sus-tainable Development which has cautioned that whilst a ‘framework for organising theselection and development of indicators is essential… any framework on its own is animperfect tool for expressing the complexities and interrelationships encompassed bysustainable development’ (cited in EC, 2005). PREST (2002) associate these problems,at least in part, to ‘our limited knowledge of the dynamics of the research area.’

These inherent difficulties are reflected in the paucity of evaluation methodologies thatfocus explicitly on sustainable development, as opposed to one or two of its facets (no-tably environmental impact). The extensive sustainability indicators that have been de-veloped at the EU level nevertheless provide robust guidelines that reflect current scien-tific understanding, cover the full range of sustainability impacts, and are consistent withEU objectives. When adapted in line with the internal terms of reference of the FP5 andFP6 objectives, client requirements and the views of stakeholders, they can be consid-ered as a useful basis of a methodology for evaluating sustainable development.

13/61

13

3 SUSTAINABLE DEVELOPMENT EVALUATION METHODOL-OGY

3.1 Evaluation process

This section presents the steps of the proposed evaluation methodology. The evaluationprocess is illustrated in Figure 1.

The methodology includes the following steps:

1. The evaluation begins with the definition of research projects to be included, i.e.screening of FP5 and FP6 research projects related to sustainable development.This task receives inputs also from METRONOME WP2 and the participants of theMETRONOME Steering Group and the Cluster Meetings.

2. After the projects are selected, the next step is necessary to define the procedureto obtain information to conduct the evaluation. The selection of data sources tobe used has been grouped in three categories: information from Final reports,data from questionnaires sent by email to project coordinators and researchers,and detailed interviews to complement the information from the questionnaires.

Figure 1. Outline of the evaluation methodology.

3. The evaluation of the contribution of projects to SuD consists of two complemen-tary approaches:

14/61

14

a. The project review matrix. This approach represents evaluation of thecontribution of projects to Sustainable Development in terms of SuD ob-jectives achieved and also SuD impacts brought about. This task is carriedout with the help of a matrix “project results vs. SuD objectives”, in whichthe information from project Final Reports is used. Performance scores foreach of the projects are defined in terms of the achievement of the corre-sponding objectives and impacts.

b. The survey. This approach represents evaluation of the use (and usability)of knowledge produced within FP research projects. As described in Sec-tion 2, the evaluation of the direct contribution of research projects toSustainable Development is a difficult task. Here, we propose that evalu-ating the quality of the research in terms of its scientific, end user, socie-tal and management and co-ordination impacts can provide a proxy of thefinal contribution to Sustainable Development. Co-ordinator questionnairesand interviews provide the main tools for this approach. Usability scoresfor each of the projects are defined in terms of the contribution withineach impact group.

4. In the final step, based on the results from the performance scores and the us-ability scores, as well as other findings obtained from the implementation of themethodology, a set of conclusions and recommendations is presented.

3.2 Selection of FP5 and FP6 projects and data collection

In the first step of the evaluation, the final list of projects to be evaluated in WP4 is de-fined. The preliminary list of projects was produced in WP2. This fine-tuning of the listincludes the verification of their suitability for the SUD approach, the availability of re-search reports in the corresponding internet sources, and the verification of their com-plete finalisation, necessary for the implementation of the METRONOME methodology.

In the second step, the data collection methods are defined. In the METRONOME pro-ject, the three following methods are used to provide data into the evaluations.

Information from Final reportsThis information is used in order to complete the matrix of project results vs. SuD objec-tives. FP projects typically result in the publication of a wide range of deliverables andoutputs, both formal and informal, and it was not considered feasible for each of thesedeliverables to be evaluated. The reviews were therefore based upon the published finalproject reports. In practice the content of the final project reports vary, although theyshould give an accurate and comprehensive overview of project research activities andtheir outcomes.

Information from questionnairesAdditional information from research projects is obtained from a questionnaire designedin WP4 in close coordination with WP2 members and with the contribution from themembers of the Steering Committee and discussions in the METRONOME Work Shop inBrussels in June 2007. The design process of the questionnaire stems form the definitionof SuD impact indicators.

15/61

15

Information from detailed interviewsIt was decided that additional information directly obtained from more in-depth inter-views could be valuable in order to complete the evaluation. For this purpose, an exten-sion of the questionnaire was designed.

3.3 Evaluation Matrix: project results vs. SuD objectives

The development of the project evaluation matrix comprised of a few distinct steps, eachof which is detailed below.

3.3.1 Identification of research objectives

The first stage of the process was the identification of the research objectives relevant toeach of the projects that were selected for evaluation. This involved a review of the FP5and FP6 research and commissioning structures.

The review identified that three levels of objectives are relevant to many of the transportresearch projects commissioned under these Framework Programmes. These are:

Specific ‘Work Programme’ or ‘Thematic Area’ objectives;Key Action (or equivalent) objectives of the Work Programme that the projectwas commissioned under; andStrategic project objectives.

In the case of projects commissioned under the FP6 IST programme an additional levelobjectives was identified, as this Priority Thematic Area is structured slightly differently tothose that other projects reviewed were commissioned under.

The first two levels of objectives were identified from the Transport Research KnowledgeCentre (TRKC) online resource (http://www.transport-research.info/web/), which iscommissioned by DG-TREN. There are a range of official resources that detail the objec-tives of each Work Programme, although for consistency all of the objectives containedwithin the matrices were sourced from the TRKC, which contains information abouttransport research programmes from across the European Research Area.

A number of the Work Programmes and Key Actions (and their equivalents) are directlyrelated to Sustainable Development (SuD), and so all of their objectives are relevant tothe review of FP5 and FP6 projects for their contribution to SuD. Many of the objectivesthat research projects have been commissioned under are not, however, directly relatedto SuD, and owing to the scope of the methodology these objectives were excluded fromthe matrices.

Table 1 below uses the FP5 GROWTH Work Programme objectives to show the types ofobjectives that were included and excluded in project evaluation matrices. While someof the objectives that were excluded could be considered to have an indirect impactupon the Sustainable Development of the Community, full or partial fulfilment of theseobjectives was not thought to give a good indication of the extent to which a transportresearch project contributes to SuD.

16/61

16

Table 1: Strategic objectives of the FP5 GROWTH Work Programme.

Strategic objectiveRelevance as an in-

dicator of a project’scontribution to SuD

To produce, disseminate and use the knowledge and technologiesneeded to design and develop processes and produce high quality,environment- and consumer-friendly products which will be competi-tive on tomorrow's market.To help increase economic growth, maintain and/or create new jobsin Europe.To sustain the continuing innovation and modernisation efforts ofmanufacturing, processing and services enterprises (including SMEs)so as to improve their competitiveness.To support the development and implementation of Community poli-cies that enable competitive and sustainable development.Development of related services (including transport) which are eco-nomic, safe and protective of the environment and quality of life.Development of quality materials, reliable measurement and testingmethods.

The optimal use of specific research infrastructures.

The objectives of all of the relevant FP5 and FP6 Work Programmes, Key Actions andtheir equivalents were tabulated and a systematic review of these objectives was con-ducted to identify which should be included in the evaluation matrices. The decisionsmade were then validated by an expert within the Centre for Sustainability (C4S) at TRL.

The strategic project objectives were identified using the RTD-Projects Database on theCommunity Research and Development Information Service (CORDIS) (http://cordis. eu-ropa.eu/). The completeness and quality of the project information found within thisproject database, which contains details of RTD projects financed wholly or partly by theEC, meant that in some instances objectives, were instead obtained from the official pro-ject websites of the selected projects. The source of the strategic project objectives (ei-ther from CORDIS or project websites) is specified on each evaluation matrix.

The evaluation matrix for each project also contains wider European sustainable devel-opment policy objectives. These are objectives that have been identified in WP2 duringthe project selection phase and are wide-ranging in their scope. The projects were alsoreviewed against three of the four indicator groups identified by WP2 (see below).

3.3.2 Impact indicators

WP2 of the METROME project had identified earlier the following indicator groups thatprojects could be reviewed against to determine their impacts.

Scientific impact indicators (these indicators explore the quality and validity of re-search projects);Customer/end-user impact indicators (these reflect the benefit of the research re-sults to end-users);Societal impact indicators (these reflect the impact of the research to society atlarge); and

17/61

17

Management and co-ordination impact indicators (referred to as co-ordination in-dicators within the evaluation matrices these ‘enabling factors’ complement theimpacts measured in the above three categories).

A number of these impact indicators were included within the evaluation matrix. Thesewere added to the matrix to provide a more general overview of the impacts of the re-search projects, and also to indicate the specific ways in which transport projects havecontributed to SuD in Europe.

The first category of impact indicators (scientific) was not included in the project evalua-tion matrices. This is because for the purpose of the evaluations it was considered thatan evaluation of the extent to which research projects had fulfilled the specific objectivesthat they had been commissioned under (as detailed previously) would give a sufficientindication of a project’s scientific impact upon SuD.

3.3.3 Evaluation matrix

The research objectives and impact indicators form the basis of the evaluation matrices,for which a number of different approaches were trialled. Two of the approaches devel-oped were considered to be fit for purpose, and so an FP6 project was selected to trialthe two approaches against. The pilot demonstrated that one of the approaches wasconsiderably more effective than the other. A skeleton template for the approachadopted is shown in Figure 2. below.

This matrix supports a qualitative evaluation of the extent to which FP5 and FP6 researchprojects have contributed to SuD. It enables reviewers to specify whether each of theobjectives have been fulfilled fully, partially, indirectly or not at all. This will result in amore comprehensive evaluation in which evaluators can make the valuable distinctionbetween whether projects that have contributed to SuD related objectives have done sofully, partially or indirectly. In practice the performance of research projects in relationto impact indicators is likely to be more clear cut, with impacts either having been real-ised or not.

It was decided that each completed evaluation matrix should be accompanied by a tex-tual summary. The summary should supplement the matrix by detailing other relevantand specific information about projects and/or their outcomes.

FP projects typically result in the publication of a wide range of deliverables and outputs,both formal and informal, and it was not considered feasible for each of these deliver-ables to be evaluated. The reviews were therefore based upon the published final pro-ject reports. In practice the content of the final project reports vary, although theyshould give an accurate and comprehensive overview of project research activities andtheir outcomes.

The quality of final reports was found to vary considerably, and so in certain instancesadditional sources of information were used to inform the project reviews. These werelimited to details contained within formal deliverables and in some instances project dis-semination reports. The two reviewers ensured that even when additional sources wereconsulted projects were evaluated using the same level and amount of information. Thisserved to retain the comparability and consistency of the reviews.

18/61

18

Fully/yes Partially Indirectly Not at all/no

Wor

kpro

gram

me

obje

ctiv

es

Soci

etal

imp

act

indi

cato

rs

Co-

ord

inat

ion

indi

cato

rs

Extent to which the project met/contributedto objectives

Cust

om

er/e

nd-

use

r im

pact

indi

cato

rs

Key

actio

nob

ject

ives

Stra

teg

icp

roje

ctob

ject

ives

Euro

pea

nSD

pol

icy

obje

ctiv

es

Figure 2. Evaluation matrix.

The majority of the projects reviewed were evaluated by two separate researchers. Thedecision to conduct two reviews served to reinforce the effectiveness and transparencyof the methodology. This was demonstrated by the fact that both of the reviewers ratedeach of the projects’ contribution to SuD almost identically. On the few occasions wherethere was a discrepancy in the reviews reasons for scoring were discussed and a consen-sus reached. Subjectivity will be inherent in any evaluation framework that evaluates thecontribution of research projects to SuD. The high level of consistency between theevaluations conducted by the two researchers does, however, indicate that the approachused has designed subjectivity out to as great an extent as possible.

The projects reviewed were mostly the same projects that questionnaires have beencompleted for. Hence, the projects were reviewed to both validate the methodology andto identify the contribution of a selection of FP5 and FP6 projects to SuD (thereby sup-plementing the information received in the questionnaires).

19/61

19

3.4 Evaluation of transferability of knowledge

3.4.1 Impact indicators

As outlined in Section 3.1- Evaluation Process- the evaluation of projects is based on theassumption that the contribution of research projects to sustainable development objec-tives depends on the extent the scientific knowledge generated is delivered to the ap-propriate target groups. In this sense, the scientific knowledge included in the Final Re-port of the projects starts a complex ‘transferability mechanism’ after it is published, sothat it is finally transferred into impacts on SuD. In this mechanism, different actors,such as researchers, end-users, policy makers, and the whole society are involved.

This transferability is measured in terms of the impact indicators groups. According tothe scheme of METRONOME Impact Model (Figure 3), the “path” followed by researchknowledge can be followed via the evaluation of the four SuD impact indicator groups.These indicator groups have been described in the following section.

Figure 3. Metronome Impact Model.

3.4.2 Data for impact evaluation: the questionnaire

In order to obtain information of the impacts of the selected projects in each of the fourindicator groups, a tailored questionnaire was designed. WP4 reviewed the existentmethodologies to evaluate projects taking into account four views: scientific perform-ances, utility for customer/end-users, societal impacts of the project and managementcoordination issues. So, the final questionnaire was composed of four parts according tothe above views. Further, indicators were identified to describe the impacts within eachof the groups (see Table 2).

20/61

20

The list of questions to be included in the questionnaire was conducted in close coopera-tion with WP2 members, which provided valuable feedback on the indicators to be usedand the text to be included in the corresponding questions. After a comprehensive proc-ess in which different drafts of the questionnaires were trialled, a 19-item questionnairewas designed.

The questionnaire uses a qualitative Likert scale with a 5 point scale (1-5). The Likertscale is the most used scale in survey research. This technique is a type of psychometricresponse scale often used in questionnaires in which respondent specify their level ofagreement to the statement (Likert, 1932). The format of the typical five-level Likertitem is as follows:

1. Strongly disagree2. Disagree3. Neither agree nor disagree4. Agree5. Strongly agree

On the other hand, Likert scaling is a bipolar scaling method, measuring either posi-tive or negative response to a statement. Sometimes a four-point scale is used; thisis a forced choice method since the option “neither agree nor disagree” is not avail-able. In our case we used a six-point scale, since the option “don’t know” is alsoavailable. There is the risk not getting answers since respondents may tend to usenothing-telling “don’t know” or “neither agree nor disagree”, which involves a kind ofacquiescence bias.

The six point scale used was as follows:

1 Completelydisagree

2 Partiallydisagree

3 Neutral4 Mostly

agree5 Completely

agreeDon’t know

The complete list of indicators and the corresponding question is included in Table 2. Thetable presents also the weights we have assigned to each indicator. The weights wereselected after sensitivity test, which is presented more detailed in Annex I. Here eachgroup of indicators is weighted by 25% of total points. The next table show the weightsfor each question, and the correspondent group of indicator in the questionnaire.

Table 2. Correspondence between indicators and items of the questionnaire and weights.Number Group Indicator Question Weight

1 Scientific Level of definition of re-search goals

The research goals requiredspecific elaboration at the

start of the project0,05

2 ScientificLevel of theoretical diffi-culties in the definition of

the methodology

There were theoretical diffi-culties in defining the re-

search methodology0,05

3 Scientific Level of Achievement ofresearch objectives

The research objectives wereall met 0,05

21/61

21

4 ScientificFitness of project re-sources to the projectneeds/expenditures

The research budget andhuman resources available

were insufficient0,05

5 Scientific Level of publication ofresults

The project results have beenadequately published in sci-entific journals and/or books

0,05

6 Utility Transferability into policyinitiatives

The project results have beentransferred into policy initia-

tives, recommendationsand/or regulations

0,04

7 Utility Fitness between end userneeds and results

Needs and views of end-users were taken into consid-

eration0,04

8 Utility Involvement level of civilservant were involved

Civil servants and/or policymakers were involved in the

project0,04

9 UtilityInvolvement level of

transport operators wereinvolved

Transport operators or ser-vice sector were involved in

the project0,04

10 UtilityInvolvement level of

transport industry wereinvolved

Transport industry sector wasinvolved in the project 0,04

11 Utility Encouragement of poten-tial for future research

The project raised new un-solved research questions 0,04

12 Utility Quality of the dissemina-tion of results

The project results have beenadequately disseminated to

end-users0,04

13 Societal Quality of disseminationthrough the website

The project webpage wasuser-friendly and updated

regularly0,08

14 SocietalLevel of encouragementreceived by society from

the project

The project encouraged theparticipation of society in re-

search (development ofawareness campaigns, public

inquiries, etc.)

0,08

15 SocietalThe extent to which the

project produced a helpfulnetworking

The project (consortium) hasimproved networking be-

tween researchers and pub-lic/private organisations

0,08

16 Management Level of stability of thenetworking

The consortium membershave gathered a stable re-

search network0,06

17 ManagementAdequacy of the fre-

quency of project meet-ings

The project included toomany consortium meetings

and Workshops0,06

18 ManagementAdequacy of the output of

the project in terms ofthe extension of Reports.

Additional effort should bemade to reduce the extension

of project Deliverables0,06

19 Management Adequacy of the financialinstrument

The financial instrument wasadequate for the project 0,06

22/61

22

In addition, the information from the questionnaire was completed from that from de-tailed interviews, in a qualitative way. This information can provide additional insights inorder to draw conclusions on the impacts of the selected projects.

3.4.3 Data analysisThe data analysis includes three separate analyses, namely: project sample characteris-tics, impact analysis and sensitivity analysis.

Project characterizationFirst, a descriptive analysis of the information of the screened projects needs to be car-ried out. This information should include data such as if the project is included in FP5 orin FP6, transport mode, spatial scale, financial instrument, EC contribution, etc.

Impact analysisThe basic source of information for the impact analysis is the questionnaires. The re-ceived questionnaire answers were analysed individually and by each impact group (sci-entific, social, utility and management) using Likert scale, which is often called a summa-tive scale.

Sensitivity analysisThe integration of results from the different items of the questionnaire depends on theweights attached to each of the items. In order to assess the influence of these weightsin the evaluation, a sensitivity analysis was carried out, in which different sets of weightsare used, and the evaluation results are then compared. The specification of scenarios isincluded in Annex I

23/61

23

4 METHODOLOGY IMPLEMENTATION

4.1 Project screeningStarting from the list of projects screened by WP 2, and as a key part of its methodology,WP4 made a first project classification according to Funding scheme: IP, STREP, NoE,etc.; scope: urban/interurban; passenger/freight; modal/multimodal; availability of pro-ject data; country of the project coordinator and consortium members, including a shortdescription.

For this task, as much information about projects as possible was collected. Then, theywere checked against the sustainable development policy objectives from the list pro-vided by WP2, and those that better met the objectives (in a scale from - to ++) leastshould have +) were finally selected, being rejected otherwise (see list and matrix be-low).

PROJECT ACRONYM PROJECT ACRONYMSUMMA PISaARTS_Rural CITYMOBILD2D FLAGSHIPSIMTAG INMARESAFET HERCULESFURORE_BTL MARSTRUCTVOYAGER SAFEINTERIORSPROGRESS SHIPMATESECOCITY CLEANENGINEPROMPT EFTCoRIMPRINT_EUROPE ECOPAINTTOSCA INTEGRATIONROSEBUD SMOKERMENSILENCE ROLLOVERHISMAR

Figure 4 Preliminary list of FP5 and FP6 selected projects.

From the list above, 16 projects were funded under FP5 and 13 under FP6, which seemsto be a good balance between both FP.

The projects were also classified according to the following four aspects (see the matrixbelow):

1. Mode: road, rail, marine, multimodal and walking.2. Scale: interurban and urban.3. Type of users: passenger, freight and passenger/freight4. Financial Instrument

24/61

24

N o ACRONYM DESCRIPTION Key words Category-Urba n,Interurban-

Category-Passenger,Freight

Mode Progra mme Ac ronym

1

23

45

67

89

10

11

121314

15

17

181920

21

222324

2526

27

2829

16

Figure 5. Review of projects: matrix by themes.

Projects 1 2 3 4 5 6 7 8 9 10 11 12 13 14Sustainable use of resources, diversifiedenergy for transport, energy-efficiency ++ +

+ - - - ++ - - ++++ - + - ++

Efficient and harmonised use of pricingmeasures, e.g. taxation and congestioncharging

- - - - - - - ++ - + ++ - + -

Reductions in transport emissions ++ - - - - - - - ++ - - - - -High-quality public services forsustainable transport ++ +

+ + - - ++ - - ++ - - + +

Developing, integrating and managing amore efficient, safer and environmentallyfriendly transport system to provide user-oriented, door-to-door services forpassengers and freight

- - ++ + ++++++ - ++ - - ++++++

Breaking the link between economicgrowth and increased traffic, and toencourage more sustainable use of thetransport system

+ + - - - - + - - + - ++ - -

Social cohésion, improved safety andsecurity ++ +

+ - ++++ - ++++ - ++ - + ++ +

High quality of urban environment ++ - - - - ++ - - ++++ - - - ++Limiting climate change - - - - - - - - + - - - - +

Figure 6. Performance of projects against European policy objectives (sample of projects).

Figure 6 illustrates the evaluation of relevance of the projects against the SuD objectives(those in the row). For instance, project number 1 in the matrix has 2 points regardingsustainable use of resources, but 0 for efficient use of pricing measures. So the projectshould be tested against the first one (sustainable use of resources), but not against useof pricing.

25/61

25

After this first selection, we noticed that some projects were still on going (Silence,Modurban and Safeinteriors, to be precise), so they were rejected from the evaluation.Results showed that 20 projects passed the test and, in order to improve the sample,WP4 included several other projects that were not in the initial list, i.e., PROSPECTS,TRANSFORUM, TRANSECON, and STEPS, and giving in the questionnaire an opportunityto the potential respondent to state “Other”.

4.2 Data collection

The data collection is partitioned into three parts: questionnaires, interviews and finalreports (online documents). Those data source were used to assess the performance ofprojects. Following chapters presents the results of the evaluation from two approaches:performance matrix and transferability of knowledge.

4.3 Results from the project evaluation matrix

At the end, 11 projects were evaluated from framework programs FP 5 and FP 6. Thebasic evaluation process concerned all individual objectives and indicators for each pro-ject separately. The valuation of a group of objectives or impacts was calculated as theaverage of the indicators in that group. Results have further been calculated to “group’saverage” in column grand total by calculating the average across the projects. Eventhough we are dealing with (just) 11 projects, in statistics, the average is an importantdescriptive measure suitable to compare groups. Also it is important to present other de-scriptive statistics as dispersion, deviation and variance which are shown below.

The Likert scale used in the project evaluation matrix against objectives is composed offour options: Fully (4), partially (3), indirectly (2) and not at all (1). But, in order to ob-tain standardized results in comparison with evaluation against impact indicators (inwhich the range is from 1 to 5), the range has been extrapolated from 1-4 to 1-5. Re-sults are shown on following tables as average values of each group of objectives as-sessed (see Section 3.3 for the definition of groups of objectives).

Table 3. Summary of Average of Objectives by Framework Program (Scale 1-5)

FrameworkProgram

EuropeanSustain-able De-

velopmentpolicy ob-jectives

Keyaction

Programmesubdivision

objective

Strategicproject

objectives

Work pro-gramme

objectives

GrandTotal

FP-5 2,88 3,75 5,00 4,80 4,06 3,89FP-6 2,19 1,56 3,18 3,61 3,44 3,43Grand Total 2,63 3,13 3,57 4,41 3,88 3,72

26/61

26

Table 4. Summary of Average of Objectives by Financial Instrument (Scale 1-5).FP FINANCIAL Total

FP5 Accompanying measures 4,04Research and technology development projects 3,73Cost-sharing contracts 3,85Thematic Networks, Concerted Action 4,00

Total FP5 3,89FP6 Coordination action 3,59

Integrated project 2,83Networks of Excellence 3,66

Total FP6 3,43Total general 3,72

In general, all the financial instrument points were greater than the central point (2.5),within a range from 1 to 5 those are acceptable results. The main differences were be-tween Integrated Projects (2.83 as average) and accompanying measures (4.04 as aver-age).

Table 5. Analysis of scores obtained in the matrix (objective) evaluation (Scale 1-5).FP Statistic Tip. Error

Average 3,8914 ,10285

3,6721Confidence interval95% 4,0779Average to 5% 3,9766Median 5,0000Variance 1,976Std. Dev 1,40558Minimum 1,25Maximum 5,00Range 3,75Amplitude intercuartile 1,25Skewness -,941 ,176

5

Kurtosis -,600 ,351

Average 3,4302 ,12501

3,2922Confidence interval95% 3,7860Average to 5% 3,4641Median 3,7500Variance 2,619Std. Dev. 1,61837Minimum 1,25Maximum 5,00Range 3,75Amplitude intercuartile 3,75Skewness -,322 ,192

Points

6

Kurtosis -1,638 ,381

27/61

27

The Table 6 shows the statistics obtained from the evaluation matrix of the EuropeanPolicy Objectives, within a range of 1 to 4, since there are calculated from directly fromthe matrix.

Table 6. Statistics for European SuD Objectives (Scale 1-4)Objectives N Minimum Maximum Media Std. Dev.

Sustainable use of re-sources, diversified en-ergy for transport, energyefficiency.

11 1 3 2,00 0,89

Efficient and harmoniseduse of pricing measures,e.g. taxation and conges-tion charging.

11 1 3 1,55 0,82

High-quality public ser-vices for sustainabletransport.

11 1 3 1,36 0,67

Developing, integratingand managing a more effi-cient, safer and environ-mentally friendly transportsystem to provide user-oriented, door to door ser-vices for passengers andfreight.

11 2 4 2,91 0,54

Breaking the link betweeneconomic growth and in-creased traffic, and to en-courage more sustainableuse of the transport sys-tem.

11 1 3 2,00 0,89

Social cohesion, improvedsafety and security. 11 2 4 2,91 0,54

High quality of urban envi-ronment. 11 1 3 2,00 0,77

Limiting climate change. 11 1 3 2,09 0,83

N valid 11

28/61

28

Table 7 presents the matrix evaluation results in relation to the impact groups.

Table 7. Average per Impact Group (Scale 1-4).

ProjectEuropean Sus-tainable Devel-

opment policy ob-jectives

Co-ordinationindicators

Customer/end-userimpact indicators

Societalimpact

indicators

1 2,38 3,25 1,86 2,002 1,50 4,00 3,14 4,003 1,50 4,00 2,29 4,004 2,88 4,00 3,14 3,335 1,50 4,00 2,71 4,006 2,00 4,00 2,86 4,007 2,13 4,00 3,14 4,008 2,13 4,00 2,71 4,009 3,00 4,00 3,14 4,0010 2,25 4,00 2,50 3,0011 1,88 4,00 3,14 4,00Grand Total 2,10 3,93 2,80 3,67

Based upon the reviews conducted the potential contribution and impact of FP5 and FP6projects is broadly similar, with no meaningful distinction to be made between the two.

The set of objectives that were the best met were the Strategic Project objectives. Thevast majority of these were considered to have been ‘fully met,’ with the others partiallymet’. This is not a surprising finding as the Strategic Project objectives are those whichare of most direct relevance to the project. The fulfilment of higher level objectives, suchas Work Programme, Key Action and Programme Subdivision objectives, instead indi-cates whether the project is likely to have broader impacts upon sustainable develop-ment in the EU, which is why it is to be expected that these will not be as fully contrib-uted to as those that the project was commissioned under. It should also be noted that itis not possible to evaluate projects based on the extent to which they have contributedto higher level objectives. This is largely because the purpose and scope of projectscommissioned varies considerably and that the magnitude and nature of the impacts andcontributions will also vary. A review against contribution to higher level objectives does,however, provide an indication of breadth of contribution.

The low sample size of projects reviewed limits the reliability of findings, but from theprojects reviewed it is surprising that the FP5 projects were considered to have contrib-uted more to Work Programme objectives than the Key Action objectives, and that FP6projects reviewed better met Work Programme objectives than Programme Subdivisionobjectives. This is surprising because the lower level objectives (Key Action and Pro-gramme Subdivision) could be considered to be more directly applicable to the projectscommissioned.

From the above, we can conclude that when a project meets its specific objectives satis-factorily, but not the higher level (e.g. Work Program or European policy) objectives, it isbecause those deal with different issues. Summary by objectives shows that sometimesthe specific objectives and the European policy are different. The methodology devel-oped in Metronome suggests that in order to evaluate FP5 and FP6 projects, it is first

29/61

29

needed to check out if the objective set out for the project coincides with those from theEU and the Programme.

The potential impact of projects in terms of the customer/end-user, societal and co-ordination indicators used was consistently positive across the range of projects re-viewed. A full list of objectives and impact indicators that FP5 and FP6 projects were re-viewed against to estimate their potential contribution to sustainable development is pre-sented in Annex II. Also, a brief analysis on the realisation of FP5 and FP6 objectives andpotential impacts is presented.

4.4 Questionnaire survey resultsIn this section we describe the main features of the questionnaire responses. A mail sur-vey is a valuable tool for gathering data from a population that is geographically dis-persed and face-to-face interviews would be too costly and time consuming (Hikmet &Chen, 2003). Social scientists have different opinions on the acceptable response rate.Research by Goyder- even not recently (see Goyder, 1985) - reports that there is a vary-ing opinion on the acceptable response rate for mail surveys among the academicians,and this rate may range between 30 and 70%.

In METRONOME WP4, a questionnaire of 19 questions (see Annex I) was delivered bye-mail to 49 participants (researchers, project coordinators, project officers, consultantsand others4). The response rate was 51% (28 answers/54 mails) and the answers col-lected during a period of 45 days. In addition, information was gathered from a personaland “in detail” interviews with project coordinators. Those gave us a general overviewabout how researchers (project coordinators) feel about EU Framework Programmes,what they would change, what they would improve, etc.

Unfortunately and despite the efforts made trying to find as many projects’ coordinators(or EU officials involved) as possible, and even sending the questionnaires twice, we onlycould get those 28 questionnaires mentioned. This means that the sample is not largeenough to drawn sound conclusions, but we still consider it is worth to spend more timeencouraging respondents to fill the questionnaires. So, one of our most interesting con-clusions is the lack of key information availability from project coordinators.

4.4.1 Sample CharacteristicsThe first part of the analysis started with a summary of the sample characteristics, (seeTable 2). As Table 8 shows, sample size is rather well balanced for Framework Pro-gramme, FP5 and FP6. Due to this condition, the conclusions could be more fitted. Ana-lyzing the Table 8, we observe more participation from University and Research Centre,which is very reasonable since WP-4 is developed and coordinated by a Research centre.Similarly, by mode (mode of transport and type of infrastructure), there are more Road.According to the Scale the number of responses for urban projects was higher than forothers.

4 Others is referring to researchers involved in projects and people recommended by partners in-volved in METRONOME.

30/61

30

Table 8. Sample Characteristics.FP Total FINANCIAL TotalFP-5 12 Coordination action 8FP-6 15 Cost-sharing 50 1 IP 3Grand Total 28 NCT 4Area of activity Total PAaSM 1Consultancy 4 (blank) 5University 10 Study contracts, evaluation con-

tracts1

Transport industry 2 NoE 1Research Centre 10 Grand Total 28Other 2 Mode TotalGrand Total 28 Road 16Country Total Multimodal 11Austria 2 Marine 1Germany 3 Grand Total 28Greece 1 Category Scale TotalItaly 1 Urban 14Netherlands 3 Interurban 11Norway 1 No specified 3Romania 1 Grand Total 28Spain 7United Kingdom 5Ireland 1Belgium 2Finland 1Grand Total 28

4.4.2 ResultsThis section is based on survey results; it is focused on the total points for each project.First, the analysis compares the total points against project features, beginning with thetotal points by project or the average when more than one observation for the same pro-ject was obtained. After, the analysis of group of indicators is presented.

The total points by project/observation have been analyzed and results are shown in thetable below. The Explore Descriptive Table provides summary statistics for Total Points,which constitutes a continuous and numeric variable. Summary statistics include meas-ures of central tendency such as the mean and the median; measures of dispersion(spread of the distribution), such as the standard deviation; and measures of distribu-tion, such as skewness and kurtosis, which indicates whether the sample comes from anormal distribution or not. Values of these statistics outside the range of -2 to +2 indi-cate significant departures from normality, which would tend to invalidate any statisticaltest regarding the standard deviation. In this case, the standardized skewness value iswithin the range expected for data from a normal distribution. Since the standardized

31/61

31

kurtosis value is within the range expected for data from a normal distribution, this vari-able (TOTAL) comes from a normal distribution.

Table 9. Explore Descriptive table.Statistics BALANCED Scenario

(Based on 100 point as highest)N 28,00Minimum 44,40Maximum 85,40Average 69,14Standard Deviation 9,71Variance 94,21

-0,61SkewnessError Skewness 0,44

0,37KurtosisError Kurtosis 0,86

Since many statistical tests assume data are normally distributed, it seems a good ideato check the distribution and, then, either transform the data or use nonparametric testthat do not require normally distributed data. The average is 69; taking into account thatthe highest could be 100 and the lowest 0, it is satisfactory points. The minimum ob-served was 44 and the maximum 85.

In the next Graph we can see a Box and Whisker plot. In descriptive statistics, a Box-and-Whisker plot is a convenient way of depicting groups of numerical data through theirfive-number summaries (the smallest observation, lower quartile (Q1), median (Q2), up-per quartile (Q3), and largest observation). A Box-and-Whisker plot may also indicatewhich observations, if any, might be considered outliers (Tukey, 1977).

Graph 1. Box and Whisker plot.

The box and whisker plot shows the median at the centre of the box and the extremesvalues as whiskers; extreme values are the maximum (85) and the minimum (44). Thebox indicates where the 50% of typical values are. Showing the outlier values as pointslocalized out of whiskers, the Graph shows one outlier, lower than 40. (Actually thereshould not be any outliers, but taking the small sample size, it cannot be avoided. How-ever, the graph gives a good overall picture of the analysis). The following tables presentthe main result scores from the questionnaire survey.

32/61

32

Table 10. Average by Impact Group and Framework Programme (Scale 1-5)FP

Scientific Utility Societal ManagementBalancedScore (1)

BalancedScore (2)

5th 3,48 3,28 3,26 3,21 3,34 3,356th 3,67 3,51 3,55 3,41 3,52 3,52

Grand Total 3,58 3,40 3,41 3,31 3,44 3,44

Table 11 Average by Financial Instrument and Framework Programme (Scale 1-5)FP contract type Balanced

Score (1)BalancedScore (2)

Accompanying measures 3,26 3,22Cost-sharing contracts 3,12 3,18No contract type 3,79 3,69Research and technology devel-opment projects 3,40 3,42

Study contracts, assessment con-tracts 3,53 3,36

5th

Thematic Networks, Concerted Ac-tion 3,84 3,88

Total 5th 3,34 3,35Coordination action 3,32 3,33Integrated project 3,89 3,83Networks of Excellence 3,42 3,50

6th

Specific Targeted Innovation Pro-ject 3,79 3,77

Total 6th 3,52 3,52Total general 3,44 3,44

In general all projects show points higher than 2.5, the central point. Taking into accountthe range is until 5, those are acceptable points. The main differences occur between theintegrated projects (3.89 from the scenario 1, 3.83 from the balanced scenario 2) andcost-sharing contract (3.18 from the scenario 1, 3.12 from the balanced scenario 2).

Table 12 Average by Scale of the project (Scale 1-5)Scale Total

Urban 3,43Interurban 3,46Grand Total 3,45

Table 13 Average by Transport Mode (Scale 1-5)Transport Mode Total

Road 3,37Marine 3,99Multimodal 3,53Grand Total 3,46

33/61

33

Table 14 Statistics by FP and Impact Group (Scale 1-5)Scientific Utility Societal Management

FP Statistic Tip.Error Statistic Tip.

Error Statistic Tip.Error Statistic Tip.

ErrorAverage 3,43 0,24 3,22 0,31 3,17 0,25 3,22 0,19

UpperLimit 2,90 2,55 2,62 2,80Confidence

interval95% Bottom

Limit 3,95 3,90 3,71 3,65

Average to 5% 3,44 3,24 3,24 3,20Median 3,78 3,50 3,33 3,25Variance 0,69 1,12 0,74 0,45Std. Dev 0,83 1,06 0,86 0,67Minimum 1,75 1,71 1,00 2,33Maximum 4,80 4,43 4,00 4,50Range 3,05 2,71 3,00 2,17Amplitude intercuar-tile 1,18 2,18 1,17 0,94

Skewness -0,63 0,64 -0,50 0,64 -1,55 0,64 0,37 0,64

5

Kurtosis 0,29 1,23 -1,43 1,23 2,95 1,23 -0,48 1,23Average 3,71 0,15 3,54 0,20 3,60 0,17 3,38 0,11

UpperLimit 3,40 3,11 3,24 3,15Confidence

interval95% Bottom

Limit 4,02 3,98 3,96 3,62

Average to 5% 3,70 3,54 3,63 3,36Median 3,50 3,43 3,67 3,50Variance 0,32 0,62 0,42 0,18Desv. típ. 0,56 0,79 0,64 0,42Minimum 3,00 2,29 2,00 2,75Maximum 4,50 4,86 4,67 4,50Range 1,50 2,57 2,67 1,75Amplitude intercuar-til 1,00 1,45 0,67 0,50

Skewness 0,19 0,58 0,21 0,58 -0,70 0,58 1,07 0,58

6

Kurtosis -1,72 1,12 -0,94 1,12 1,72 1,12 2,59 1,12

34/61

34

4.5 Integration of assessment results

To integrate, but on the other hand compare the results from the project evaluation ma-trix and the survey, a multiple criteria analysis was developed.

Estimation of the total scores: The total score were produced from weights of balancedscenario and the Likert scale detailed below.

Multi-criteria assessment: We used a multi-criteria methodology similar to Flag Model(See Nijkamp and Ouwersloot 1997) in which the values obtained in each indicator arecompared with reference values called Critical Threshold Values (CTV). Each indicatorshould have a CTV because is indicator belongs to a group or category.

Estimation of the Critical Threshold Values: The scale from 1 to 5 was used for bothevaluations Knowledge Transferability Score and Performance Score. Altogether the scalewas composed of 5 thresholds: scale minimum and maximum and the quartiles 25%,50% and 75%. This was done to analyse the performance level in absolute scale sepa-rately for FP5 projects and FP6 projects, and to compare the results. For comparing pro-jects to each other in more detail another method to define the thresholds could havebeen used: set the maximum obtained value of the group as the maximum threshold(100%) and likewise the lowest value obtained as the minimum threshold (0%), andthen define the quartiles 25%, 50% and 75% using the new scale of each group. How-ever, for clearness of the methodology only the absolute scale was used.

Graph 2. Box and whisker plot by Indicator Group and FP.

35/61

35

Analysis of total points by groups of indicators: Category is defined by arrows and col-ours, which are explained as follows.

Green: the project achieved more than 75% of the maximum.Yellow-up: the project achieved points between 50% and 75%Yellow-down: the project achieved points between 25% and 50%Red: the project achieved fewer points than 25%

Analysis of symbols: The most important issue in this kind of presentation is that the col-our and type of arrow is only related to the category achieved of the project by using thethresholds defined in the multi-criteria analysis being thus independent of the originalscale. However, in this case when the original scale was used the symbols show also theactual rating and points achieved.

This integration exercise provides an example of how the overall assessment of FP5 andFP6 projects could be conducted. An extension of this exercise with a larger number ofprojects and data collection resources could provide additional insights, which could bevaluable to complement the implementation of methodological framework suggested inthis Deliverable.

36/61

36

Table 15. Comparing Scores: Knowledge Transferability Score and Performance Score (Scale 1-5), CTVs: 0-1.25-2.50-3.75-5.00.

Projects Key actionProgrammesubdivisionobjective

Workprogrammeobjectives

EuropeanSustainable

DevelopmentPolicy objectives

Strategicproject

objectives

Co-ordinationindicators

Customer/end-user impactindicators

Societalimpact

indicatorsManagement Scientific Utility Societal Balanced

Score

1 5,00 5,00 4,17 2,97 5,00 4,06 2,32 2,50 4,00 2,78 3,00 2,50 3,182 4,06 1,88 5,00 5,00 3,93 5,00 3,50 3,20 3,43 4,00 3,503 1,88 1,88 5,00 2,86 5,00 3,00 3,00 3,29 3,33 3,174 4,17 5,00 3,59 5,00 5,00 3,93 4,17 3,58 3,87 3,48 3,33 3,565 2,50 3,33 1,88 4,58 5,00 3,39 5,00 2,75 2,50 1,71 3,33 2,606 3,75 4,38 2,50 4,84 5,00 3,57 5,00 3,75 4,80 3,29 3,67 3,887 3,75 3,75 2,66 5,00 5,00 3,93 5,00 3,00 4,20 4,00 4,33 3,888 1,56 3,44 2,66 5,00 5,00 3,39 5,00 3,56 3,98 2,61 3,08 3,359 3,75 4,17 3,75 4,69 5,00 3,93 5,00 2,33 3,00 4,14 2,33 3,1910 3,75 4,17 2,81 4,53 5,00 3,13 3,75 3,25 3,60 2,83 3,33 3,2611 2,68 3,44 2,34 5,00 5,00 3,93 5,00 3,06 3,43 3,56 3,42 3,30GrandTotal 3,13 3,57 3,88 2,63 4,41 4,91 3,50 4,58 3,31 3,50 3,12 3,27 3,32

Knowledge Transferability: Average by group of indicatorsImpact Indicators

Performance Score: Average by group of ObjectivesResearch Objectives

Coordination impact indicators seem to receive better scores in the Matrix evaluation against objectives than in the Questionnaire evaluation(see Management indicators). As the same occurs with societal impact indicators perhaps, some over-estimating has been occurred in the for-mer case. European Policy objectives have obtained the lowest score. In the following table the results are presented in Total Points.

Table 16. Comparing Scores: Knowledge Transferability Score and Performance Score (Scale 1-100) CTVs: 0-25-50-75-100.

Projects Key actionProgrammesubdivisionobjective

Workprogrammeobjectives

EuropeanSustainable

DevelopmentPolicy objectives

Strategicproject

objectives

Co-ordinationindicators

Customer/end-user impactindicators

Societal impactindicators Management Scientific Utility Societal Balanced

Score

1 100 100 83 59 100 81 46 50 80 56 60 50 642 81 38 100 100 79 100 70 64 69 80 703 38 38 100 57 100 60 60 66 67 634 83 100 72 100 100 79 83 72 77 70 67 715 50 67 38 92 100 68 100 55 50 34 67 526 75 88 50 97 100 71 100 75 96 66 73 787 75 75 53 100 100 79 100 60 84 80 87 788 31 69 53 100 100 68 100 71 80 52 62 679 75 83 75 94 100 79 100 47 60 83 47 6410 75 83 56 91 100 63 75 65 72 57 67 6511 54 69 47 100 100 79 100 61 69 71 68 66Grand 63 71 78 53 88 98 70 92 66,27 70,08 62,37 65,40 66,44

Performance Score: Average by group of ObjectivesKnowledge Transferability: Average by group of indicators

Research Objectives Impact Indicators

37/61

37

4.5.1 Integrated Analysis by Financial Instrument

As Table 17 below shows, every Financial Instrument obtained acceptable score, within arange of 1-5. The lowest score was 2.83, which still is higher than the middle point ofthe range 2.5. Also, the graph shows the average line for objective evaluation, in 3.67(data from evaluation matrix of objectives) and the average line for indicators evaluationin 3.50 (data from survey). Satisfactorily all average values of Financial Instrument cate-gories are fairly close to those total average lines (difference 10% at the maximum) ex-cept the Integrated Project in the case of Objective Average. This means that regardingthe financial instruments used in both FP5 and FP6 there are no major differences intheir performance for fulfilling SuD goals.

Graph 3. Average score of Indicators (knowledge transferability) and Objectives (perform-ance) by Financial Instrument and FP (Scale 1-5).

Table 17. Average score of Indicators (knowledge transferability) and Objectives (perform-ance) by Financial Instrument and FP (Scale 1-5)

FP Financial IndicatorsAverage

ObjectivesAverage

FP- 5 Accompanying measures 3,22 4,04Research and technology development pro-jects

3,42 3,73

Cost-sharing contracts 3,18 3,85Thematic Networks, Concerted Action 3,88 4,00Study contracts, assessment contracts 3,36

FP-6 Coordination action 3,33 3,59Integrated project 3,83 2,83Networks of Excellence 3,50 3,66Specific Targeted Innovation Project 3,77

Total gen-eral

3,50 3,67

38/61

38

DISCUSSION

4.6 Methodological development

After the results from the matrix (i.e. in terms of performance score) and from the ques-tionnaires and interviews (i.e. in terms of transferability score) are calculated, some con-clusions and recommendations can be drawn from the methodological development.

Firstly, we use the findings obtained from the analysis of the above mentioned scores.These refer mainly to the different results obtained by FP5 and FP6 projects, the com-parison of results in terms of the contribution to SuD objectives or in terms of theirknowledge transferability relating SuD issues, as well as additional conclusions from de-tailed results. We also used different weightings for the indicator groups. Generally, wecan conclude that the results were not very sensitive to changes in the weights, sincethere is not much variation in the series lines of results for the different weights used.Although projects’ ranking had different behaviour in each scenario the lines did not fluc-tuate considerable.

Secondly, from the methodological point of view the exactly same projects should be in-cluded in both types of analyses (matrix evaluation and questionnaire) carried out thussupplementing each other i.e. all those projects that questionnaires have been com-pleted should also be reviewed for their contribution to SuD objectives. This would havebeen ideal for both to validate the methodology and to identify the contribution of a se-lection of FP5 and FP6 projects to SuD. Unfortunately this was not always the casemainly for two reasons, partly because the two processes were running simultaneouslyand we could not know from which projects we would get the answers and partly be-cause it was not possible to find documents from all projects.

FP projects typically result in the publication of a wide range of deliverables and outputs,both formal and informal, and it was not considered feasible for each of these deliver-ables to be evaluated. The reviews were therefore based upon the published final pro-ject reports. In practice the content of the final project reports vary, although theyshould give an accurate and comprehensive overview of project research activities andtheir outcomes. As the content and quality of final reports was found to vary considera-bly, in certain instances additional sources of information were used. These were limitedto details contained within formal deliverables and in some instances project dissemina-tion reports. The reviewers ensured that even when additional sources were consultedall projects were evaluated using the same level and amount of information. This servedto retain the comparability and consistency of the reviews.

The majority of the projects were reviewed by two separate evaluators who were re-searchers themselves. The decision to conduct two reviews served to reinforce the ef-fectiveness and transparency of the methodology. This was demonstrated by the factthat both reviewers rated all projects almost identically regarding all objectives and indi-cators of contribution to SuD. On those few occasions where there was discrepancy inthe reviews reasons for scoring were discussed and a consensus reached. Subjectivitywill be inherent in any evaluation framework that evaluates the contribution of researchprojects to SuD. The high level of consistency between the evaluations conducted bythe two researchers does, however, indicate that with the approach and design usedsubjectivity was not a problem and the results are liable.

39/61

39

Non-response rate of questionsWe tried to analyze which questions had the highest non-response rate (were less an-swered) and why. The number of “Don’t know” answers is shown in Graph 6 of Annex Iand the most important findings are the following:

The group with most “don’t know” answers was that of scientific, whilst the lesswas utility group. Especially answers to question 2: There were theoretical diffi-culties in defining the research methodology? were not known.

Some people answered, “I don’t know” because they started to work in the pro-ject after the project was launched.

Sometimes the researcher did not know about the scientific results (publications,objectives, journals or books) of the project which shows a certain lack of follow-up.

Which questions were answered with most certainty? Why?The question with the most answers “completely agree” was the number 7: Needs andviews of end-users were taken into consideration. On the other hand, question 17: Theproject included too many consortium meetings and workshops; and question 10: Trans-port industry sector was involved in the project got the most “completely disagree” an-swers. Although the sample was small we still can conclude that the perception of theamount of meetings tells that prevailing practice is good. Regarding third party involve-ment we can say that end-users’ opinions have at least taken well into consideration butthat transport industry has not strongly been involved in projects with sustainabilitygoals.

Detailed InterviewsDetailed interviews were planned to be the third, additional part of the evaluation meth-odology aiming at more detailed information about the projects, dissemination and useof the results. Unfortunately, only one out of the planned four interviews could be carriedout in reality. Taking this as a fairly typical transport project the main findings regardingthe efficiency in research and outcomes are stated in the following:

Research Methodology: The aim of the project was to measure sustainability of atransport project. It was difficult to establish the objective as transport projectscan influence several generations. “The inter-temporal allocation of utility” was animportant factor in this case. Another issue was the tri-modal concept of sustain-ability: economic, social and environmental.

Research Objectives: The research objectives needed to be adjusted to EC objec-tives. The project delivered useful results, even when further research would beneeded.

Results and Dissemination: In this case, the project had a satisfactory level ofpublications and results were used in EC-policy recommendations, although thereare no known direct transfers. In addition, there are many articles, conferencepapers and journal publications about the project results.

Participation: It was difficult to involve civil servants, but transport operatorswere involved in workshops.

Research networks: Regarding building up research networks not too much wasachieved but during the workshops quite a few contacts were made.

40/61

40

Recommendations: This project would have preferred “a higher budget and morefreedom to make changes in the original allocation of resources, more flexiblemilestones, and sub-deliverables, in order to produce similar outcomes”.

As this was just an example of how a “depth interview” should be conduced to give addi-tional information for the evaluation we only accepted those general answers that couldfit for other projects as well. Of course, there were answers that only could be specifi-cally attributed to the project in question.

Regarding the last point, it is true that, from a certain point of view, eliminating sub-deliverables could lead to an uncontrolled situation. Leaving all the project results to thefinal report has the risk to find incorrect results just at the end of the project, withouttime to repair. Regarding the time table milestones and intermediate reports tend to for-ce members to keep the schedule.

4.7 Results

4.7.1 Project performanceAs stated already before in this Deliverable, the sample of projects screened is not largeenough to draw solid conclusions regarding FP5 and FP6 transport projects contributingto Sustainable Development. However, the total evaluation points by objective groupshow clearly which objectives are difficult to meet and which easier (Table 3). Key ActionObjectives are harder to meet than the Work Programme Objectives or the ProgrammeSubdivision Objectives. European Policy Objectives are the hardest to meet probably be-cause of their very general nature. Strategic Project Objectives are best met as could bepredicted; naturally it is easier for a project to meet its specific objectives than the Euro-pean policy. Also, we can conclude that when a project meets its specific objectives sat-isfactorily, but not the European policy, it is because the goals are different. It might bethat the project is concentrated to one European policy goal only and thus has beenevaluated low for the other goals. The methodology developed in Metronome suggeststhat in order to evaluate both projects itself and a whole programme, it is first needed tocheck out if the objectives set out for the project coincides with those from the EU andthe programme.

Looking at evaluation results of European Policy Objectives separately (Table 6) we cansee that objectives regarding societal impacts, safety, security and environmentallyfriendly transport systems are better met than the others are. Regarding indicators de-scribing impacts of the projects Co-ordination Activities and Societal Impact have sub-stantially higher scores than indicators for Customer or End-user Impact (Table 15).

In all, our evaluation results show that the Performance Objectives are somewhat bettermet in FP5 than FP6. Regarding the financial instrument there are no remarkable differ-ences except Integrated Projects that performed worst (Table 17).

4.7.2 Knowledge transferabilityResults regarding Knowledge Transferability are obtained from the Questionnaire Inter-view. Using balanced weighting (see Table 2) to obtain the basic result we see that thetotal average is fairly high, 3.32 in scale 0-5 (Table 15). In addition, there is very littlevariation between the averages of the four indicator groups. Scientific indicator group

41/61

41

has the highest score of 3.50, Utility indicator group the lowest 3.12, the middle onesbeing Management and Societal groups with average points 3.31 and 3.27 respectively.

Likewise, analysing the total points by transport mode the project dealt with the averagefor Multimodal projects 3.53 is higher than that of Road projects 3.37 (Table 13), butalso the dispersion of the Multimodal sample is higher. There were too few Marine pro-jects to analyse the results separately. There was no difference in the average score be-tween urban and interurban projects (Table 12).

Framework Program 6 obtained somewhat higher total average score than FP5 (seeTable 10). The dispersion in FP5 projects is larger than FP6 projects for which the resultsare more centred. Regarding the financial instruments used in both FP5 and FP6 thereare no major differences in their performance for fulfilling SuD goals.

42/61

42

5 CONCLUSIONS AND RECOMMENDATIONS

5.1 ResultsThe literature review confirms the existence of methodological difficulties in measure-ment, valuation and comparison that has complicated the progress of evaluation of theimpacts of research upon sustainable development. However, this “lack” is currently sup-ported by practical experience, and the sustainability indicators developed at the EUlevel, provide robust guidelines that reflect current scientific understanding covering thefull range of sustainability impacts, and are consistent with the EU objectives. Theevaluation of the direct contribution of research projects to Sustainable Development is adifficult task. Hence, we propose here that evaluation of the quality of the research interms of its dissemination and transferability into practice constitute a proxy of their finalcontribution to SuD objectives. The more research carried out in a project is adequatelytransferred, the more the SuD objectives have a potential to be achieved.

The analysis of the questionnaires shows that there were quite many responses stating“don’t know”, mainly regarding those questions related to Scientific Performance indica-tors (see Annex I).

In mailing surveys an acceptable response rate is within the range of 30% and 70%. Inour survey the response rate was 51%. Although this is not a major problem regardingthe scope of Metronome, in all the number of responses and thus projects included isfairly low. We can assume some conclusions about the reasons for people not to answerthe survey:

In general, the follow-up of a project is not seen to be very important, not atleast for the partners involved but not for the project coordinator either. Thework is done when the project is finished.

People are not aware of the project results and so they do not feel capable to an-swer the survey.

In order to answer the survey it is important to know the project quite well. So,apart from the coordinator who often after 4 or 5 years from the end of the pro-ject is not any more with the same organisation, no one else knows the projectwell enough to answer.

In general, people are not aware about the later exploitation of the project re-sults after its life time and do not feel responsible of future duties.

Some detail comments:

Not many scientific networks seem to be risen, but interesting contacts havebeen made among participants in the projects.

The meeting practice is good as the number of meetings held throughout theproject life was highly agreed.

Transport industry seems not to have been involved greatly.

The final meeting and the web site have proved to be important instruments fordissemination to end-users. Therefore claiming a fixed period to keep the website“on air” after the end of the project is an important decision.

43/61

43

From both approaches evaluated, transferability of knowledge and achievementof objectives, results did not show a relationship between management score andobjectives score, which indicates that projects can be well-managed and satisfac-torily disseminated even if their objectives do not always match with general ob-jectives Sustainable Development. One conclusion is that the SuD goals (Per-formance Indicators) are not an issue of great concern. On the other hand it wasfound out that the project’s own goals were the most important ones, and theydidn’t always meet the higher level SuD objectives.

Some practicalities could be mentioned:

There were difficulties in finding information about the projects in the internet.The problem could be overcome by creating a data base with the information ofthe required final reports.

We also found difficulties in analysing the final reports to get information anddata from the projects as it is for certain that no one is likely to admit failures,gaps, etc. but tries to present the project successfully. In the questionnaire, theonly failure admitted was the lack of resources. This led us to add the option ofconducting detailed interviews with the project coordinator still guarantying theconfidentiality of the respondent.

5.2 Conclusions

5.2.1 ResultsMain conclusions from the methodology and its application:

Our first focus has been on methodological development for an evaluation meth-odology for Framework Programme performance regarding Sustainable Develop-ment. Secondly, we have applied the method to a sample of FP5 and FP6 trans-port projects. In addition to the actual results of applying the method, other con-clusions and recommendations arising from the implementation of the methodol-ogy, as well as suggestions for further research should be presented.

We used two approaches and data bases in the evaluated. The first approach wasto use and analyse the final reports and sometimes also other official documentsto evaluate the achievement of objectives set out either by EU policy, the WorkProgramme or Action and the project itself and the impacts caused. The secondapproach was to conduct a questionnaire survey directed to the coordinators orother key persons of the selected projects with the main focus on transferabilityand use of results and knowledge created in the project.

In both data collection methods used we faced severe problems. Namely, it wasvery difficult to find project documents after several years from the end of theproject as no common data base for the final reports exist. Regarding the ques-tionnaire interview carried out by email there were two main problems, at first itwas difficult to find the present email address of the respondent and secondly thepoor response rate. Although the number of questionnaires returned and ana-lysed does not allow to draw statistically confidential FP wide conclusions it couldbe used to test the methodology developed and give fairly good estimates of theresults as well.

44/61

44

5.2.2 Performance ScoreRegarding the achievement of the different SuD-related objectives results show that thehe set of objectives that were the best met were the Strategic Project objectives. Thevast majority of these were considered to have been ‘fully met,’ with the others partiallymet’. This is not a surprising finding as the Strategic Project objectives are those whichare of most direct relevance to the project. The fulfilment of higher level objectives, suchas Work Programme, Key Action and Programme Subdivision objectives, instead indi-cates whether the project is likely to have broader impacts upon sustainable develop-ment in the EU, which is why it is to be expected that these will not be as fully contrib-uted to as those that the project was commissioned under. It should also be noted that itis not possible to evaluate projects based on the extent to which they have contributedto higher level objectives. This is largely because the purpose and scope of projects com-missioned varies considerably and that the magnitude and nature of the impacts andcontributions will also vary. A review against contribution to higher level objectives does,however, provide an indication of breadth of contribution.

The low sample size of projects reviewed limits the reliability of findings, but from theprojects reviewed it is surprising that the FP5 projects were considered to have contrib-uted more to Work Programme objectives than the Key Action objectives, and that FP6projects reviewed better met Work Programme objectives than Programme Subdivisionobjectives. This is surprising because the lower level objectives (Key Action and Pro-gramme Subdivision) could be considered to be more directly applicable to the projectscommissioned.

It could, however be concluded that when a project meets its specific objectives satisfac-torily but not the European policy, it is because the goals differ from each other. It mightnot even have been the intention of the project to fulfil more than one of the Europeanpolicy goals.

The potential impact of projects in terms of the customer/end-user, societal and co-ordination indicators used was consistently positive across the range of projects re-viewed. The best scores regarding impacts of the projects evaluated from final docu-ments was given to impacts of Coordination Activities, next Societal impacts and lastCustomer/End-user impacts.

5.2.3 Transferability scoreLooking at the transferability of scientific knowledge from the questionnaire interviewresults both FP5 and FP6 projects have got scores above average level of the assessmentscale, which demonstrates that the projects have succeeded in translating the results ofthe research in the scientific, utility, societal and management domains. Within thesefour domains, scientific objectives have obtained the highest scores, closely followed byutility, societal and management domains. These results are averages across projectsusing balanced weighting of the four domains for the questions.

A closer look at the results shows that:

Framework Program 6 obtained a somewhat higher total average score than FP5which may indicate an enhanced dissemination and use of research results in FP6compared to FP5, but it can also be due to time since much less time has lapsedsince FP6 than FP5. The dispersion in FP5 projects is larger than in FP6 projectsfor which the results are more centred meaning in practice that the FP6 projectsincluded were more equal in quality than those of FP5.

45/61

45

Regarding the transport mode, multimodal projects obtained better results thanroad projects.

There were no significant differences between projects covering urban and thoserelated to interurban transport, both obtaining satisfactory results.

In addition to the transferability score values, information from the detailed interviewconstituted a productive example of how a more in-depth communication with projectcoordinator can provide additional insights for the evaluation process.

In all, the results show that the specific projects selected for the evaluation of their con-tribution to Sustainable Development have succeeded in providing such research thatcan be transferred into a practice contributing to SuD objectives.

5.2.4 Financial instrumentRegarding the financial instruments used in both FP5 and FP6 there were no major dif-ferences in their performance in fulfilling SuD goals according to the both evaluation ap-proaches used, performance evaluation and transferability evaluation.

5.3 Extensions and improvements of the proposed methodologyIn our opinion, regarding objectives of all the different actors, levels and themes therecould be a little more consistence and clearness between the objectives at the Europeanlevel and those of the specific Programme and Call with a more specific definition of thetargets to be met in the specific Call to be addressed.

For the evaluation of projects and programmes long after their lifetime it would be es-sential that the projects would be followed-up even after their official life time and allfinal results as well as exploitation and dissemination activities would be stored. Thereare several possibilities to solve the problem in practice, two possible suggestions:

A general obligation stating that after the project has been concluded, to presentan annual report for three years providing information about the number of publi-cations, research networks constituted, patents, papers published etc.

A more flexible relation between finance and output, i.e. an agreement of fi-nance, desired output and final delivery date of the final results, disseminationand exploitation reports (i.e. eliminating much of the intermediate milestones andsub deliverables).

In such a way, the Commission could develop an open database available to everyoneinterested in a project including not only the results of the project but also the follow-upfor at least for three years. This would have been exactly the data needed for Metro-nome.

Although project partners should be responsible to try to make commercial or ongoinguse for their work, the fact is that it is beyond researchers’ activity to influence EU policyfirstly, because there is no budget for these activities, and secondly, most researchersdon’t have skills or motivation to do a sales agent’s work, and they lack the contacts.The dissemination and exploitation could be easier if a wider rage of actors would be in-volved in the R&D projects, but

46/61

46

It has been proven difficult to get direct involvement from civil servants and/orpolicy makers, especially from the EC. It proves to be easier to involve wellknown policy analysts and policy advisers, who use to be much willing to partici-pate in the workshops and dissemination events.

End users are likely to be indirectly involved in the projects as participants in theworkshops rather than directly as partners.

After the end date of the project, usually EC has no involvement in the dissemina-tion activities. Apparently, the interest for the project stops at the moment of thecompletion and acceptance of the final deliverable and last financial documents.

Sometimes not a higher budget, but more freedom to make changes in the original allo-cation of resources to specific activities can provide the flexibility needed to addressmore research questions or study some issues more deeply than originally planned.

Projects should be able to establish "concerted actions" across projects during their lifetime, meet in a structured way, really exchange knowledge and develop concepts to-gether, and integrate each other's results into their project work. Also, an exchange ofmethodologies between similar or cluster projects in order to validate each methodologydeveloped could be very useful, which means the use of the same projects in the meth-odological development between the cluster projects.

As our methodology includes the questionnaire interview directed to the coordinators orother key persons of the projects we suggest the following to over come the problem ofthe high non-response rate:

To include more projects i.e. send more questionnaires (which entails more timeand resources)To offer some kind of reward to those answering the surveyTo add a follow-up task as the last task of the management work packageTo use personal contacts, a telephone interview (not always possible since tele-phone numbers are not always available) has proven to be more efficient thanemail, or intervention of a colleague knowing the interviewee (someone from thesame organisation or other relations) to help to get a response.The questionnaires should be personally addressed, and if only possible, shouldbe sent officially by the Commission to have more authority.

The very careful planning and addressing applies to any interviewee group whether it willbe project personnel, civil servants or any other end-user group.

47/61

47

References

Capron, H., van Pottelsberghe de la Potterie, B. (1997) Public support to R&D pro-grammes: an integrated evaluation scheme. In OECD (1997a) Policy Evaluation in Inno-vation and Technology: Towards Best Practices. URL:http://www.oecd.org/document/7/0,3343,en_2649_34273_1813191_1_1_1_1,00.html

European Commission (EC) (2005) Communication from Mr. Almunia to the members ofthe Commission. Sustainable Development Indicators to monitor the implementation ofthe EU Sustainable Development Strategy. SEC (2005) 161 final

EC (2006) EU Renewed Sustainable Development Strategy. URL:http://ec.europa.eu/sustainable/welcome/index_en.htm

EC (2006) Impact evaluation guidelines. SEC (2005) 791

Eurostat (2007) Analysis of national sets of indicators used in the National Reform Pro-grammes and Sustainable Development Strategies. Eurostat: Methodologies and work-ing papers

Eurostat (2007) Measuring progress towards a more sustainable Europe. 2007 monitor-ing report of the EU sustainable development strategy. Eurostat Statistical books

Goyder (1985)

Hametner, M., and Steurer, R. (2007) Objectives and Indicators of Sustainable Develop-ment in Europe: A Comparative Analysis of European Coherence. ESDN Quarterly ReportDecember 2007

Hikmet, N., & Chen, S. (2003). An investigation into low mail survey response rates ofinformation technology users in health care organizations. Medical Informatics (72), 29-34.

Kimble, M. and Tight, M. (2007) The effectiveness of national transport strategies as ameans to promote the development of more sustainable transport systems. Associationfor European Transport and contributors 2007

Langer, M., and Schon, A. (2002) An integrated referential framework for sustainabledevelopment. URL: http://www.sustainability.at/easy2002/easy_eco/pdf/langer.pdf

Langer, M., Schon, A., Egger-Steiner, M., and Hubauer, I. (2003) Evaluating Evaluationin the context of Sustainable Development (I) The Planning and Commissioning Proce-dure of Evaluations with Sustainable Development as part of a Tool Box. URL:http://epub.wu-wien.ac.at/dyn/virlib/wp/mediate/epub-wu-01_6e7.pdf?ID=epub-wu-01_6e7

Likert (1932)

Luukkonen, T. (1997) The increasing professionalisation of the evaluation of mission-oriented research in Finland: implications for the evaluation process. In OECD (1997a)Policy Evaluation in Innovation and Technology: Towards Best Practices. URL:http://www.oecd.org/document/7/0,3343,en_2649_34273_1813191_1_1_1_1,00.html

48/61

48

Mitchell, G., Gawthorpe, S., and Namdeo, A. (2005) Evaluation criteria for SOLUTIONS.SOLUTIONS discussion paper. EPSRC

Nijkamp, P., Ouwersloot, H., (1997). A decision support system for regional sustainabledevelopment: The Flag Model. Tinbergen Institute Amsterdam, Tinbergen Paper. 74/3.

Organisation for Economic Co-operation and Development (OECD) (2007) The evaluationof scientific research: selected experiences. OECD, Paris.

OECD (2002) Overview of sustainable development indicators used by national and in-ternational agencies. OECD statistics working paper 2002/2. STD//DOC(2002)2. URL:http://oberon.sourceoecd.org/vl=821436/cl=30/nw=1/rpsv/cgi-bin/wppdf?file=5lgsjhvj7p8v.pdf

OECD (2001) The DAC Guidelines. Strategies for Sustainable Development. URL:http://www.oecd.org/dataoecd/34/10/2669958.pdf

PREST (Policy Research in Engineering, Science & Technology) (2002) Assessing theSocio-economic impacts of the Framework Programme. URL:ftp://ftp.cordis.europa.eu/pub/fp5/docs/fp5_monitoring_socio-economic_final_report.pdf

Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley Reading: MA.

Transport Mistra (no date) Impact – Implementation Paths for ACTion – towards sustain-able mobility. URL:http://www.mistra.org/program/mobility/impact/home.4.58b55df710900f8463c800022822.html

United Nations (2005) Agenda 21. URL: http://www.un.org/esa/sustdev/agenda21text.htm

Widmer, T. (2002) Evaluation standards in the context of sustainability. EASY-ECO(EvAluation of SustainabilitY EuroCOnference May 23 – 25, 2002 Vienna / Austria.

49/61

49

ANNEX I

Sustainable Development Objectives

Industrial Competitiveness Sustainable Development Community and Public Poli-cies

Worldwide excellence in inno-vation and technologies ap-plied to transport domains;promoting the use of ICT as

well as take-up actions on in-telligent transport systems

(ITS)

Sustainable use of resources,diversified energy for trans-

port, energy-efficiency

Improvements in the relation-ship between research and

policy at all levels in the EU,timely and effective scientific

inputs for wide field of policies

Leadership in eco-industry,eco-innovations; development

of environmentally friendlycars and other transport vehi-cles, and new power train con-

cepts.

Efficient and harmonised useof pricing measures, e.g. taxa-tion and congestion charging

Improvements in the efficiencyand transparency of decision

making

Definition of a set of systemsarchitectures covering variousmodes of transport, differenttypes of user, communication

means and protocols

Reductions in transport emis-sions

Improved policy coherence

Rebalancing and integration ofdifferent transport modes,promoting interoperability;

developing systems for man-aging modal and intermodaltraffic and transport systems

High-quality public servicesfor sustainable transport

Expanded and improved in-vestment in human capital

Improved European infrastruc-ture, eliminated infrastructure

bottlenecks

Developing, integrating andmanaging a more efficient, sa-fer and environmentally friend-ly transport system to provide

user-oriented, door-to-doorservices for passengers and

freight

Development of the EuropeanResearch Area and scientificsupport for EU enlargement

Efficient internal market andlabour market

Breaking the link betweeneconomic growth and in-

creased traffic, and to encour-age more sustainable use of

the transport system

Better implementation of exist-ing EU environment policies

and legislation at the local lev-el

Attractive business environ-ment + entrepreneurial culture Social cohesion, improved

safety and securityGreater coherence between

macroeconomic and structuralpolicies

Efficient use of the knowledgesociety: making R & D a top

priorityHigh quality of urban environ-

mentGood governance

Limiting climate change Co-operation between minis-tries

Quantitative tools for decisionmaking, reflecting socio-

economic objectives

50/61

50

Questionnaire model

Figure. 1 Questionnaire model

51/61

51

Don’t know answersGraph 4. - Don’t know answers

0

1

2

3

4

5

6

7

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19

Scientific Performance Utility Societal Management

'Don´t Know' answer by Type of indicator

52/61

52

Sensitivity analysis process

The main objective of this section is to look at the changes in the performance indicatorsusing different weights corresponding to three types of weighting balanced, extreme andintermediate. We end up to ten scenarios as follows (see Table 18 for summary).

1. Balanced scenarios: There are two balanced scenarios constructed as follows

Scenario 1: all indicators (questions) had the same weight, 1/19, and the total isequal to 100%

Scenario 2: the total was equitably distributed in four groups of indicators (scien-tific, utility, management and societal), 25% for each, which again was distrib-uted evenly for all indicators (questions) within the group.

2. Extreme scenarios: There are four extreme scenarios constructed from each indi-cator group alone as follows:

Scenario 3-Scientific: This scenario involved only the scientific indicators group,questions 1 to 5 evenly weighted.

Scenario 4-Utility: This scenario involved only the utility indicators group, ques-tions 1 to 5 evenly weighted, questions 6 to 12 evenly weighted.

Scenario 5-Societal: This scenario involved only the management indicatorsgroup, questions 13 to 15 evenly weighted.

Scenario 6-Societal: This scenario involved only the societal indicators group,questions 16 to 19 evenly weighted.

3. Intermediate scenarios: Are combinations of the previous two types. There arefour intermediate scenarios referred to the allocation of the 50% of the totalpoints for one group of indicators; while the other half is distributed to the re-maining three indicator groups depending on the number of questions, eachquestion having equal weight.

Scenario 7- Scientific: The scientific indicator group was weighted by 50%, by25% Utility, 11% societal and 14% management.

Scenario 8- Utility: The utility indicators indicator group was weighted by 50%, by21% Scientific, 13% Societal and 17% Management.

Scenario 9- Societal: The societal indicator group was weighted by 50%, by 17%Scientific, 22% Utility and 13% Management.

Scenario 10- Management: The management indicator group was weighted by50%, by 27% Scientific, 23% Utility and 10% Societal.

The next table shows the weights and number of questions used in the hypothetical sce-narios in order to assess the Sensitivity of the results.

53/61

53

Table 18 Definition of scenarios for the sensitivity analysis

Scenarios IndicatorsWeight /

group# of Ques-

tions involvedWeights /question

Scenario 1 - 1 19 0,05

Scientific 25% 5 0,05

Utility 25% 7 0,04

Societal 25% 3 0,08

BalancedScenario 2

Management 25% 4 0,06

Scientific 100% 5 0,2

Utility 0% 0 -

Societal 0% 0 -Scenario 3

Management 0% 0 -

Scientific 0% 0 -

Utility 100% 7 0,14

Societal 0% 0 -Scenario 4

Management 0% 0 -

Scientific 0% 0 -

Utility 0% 0 -

Societal 100% 3 0,33Scenario 5

Management 0% 0 -

Scientific 0% 0 -

Utility 0% 0 -

Societal 0% 0 -

Extr

eme

Scenario 6

Management 100% 4 0,25

Scientific 50% 5 0,10Utility 25% 7 0,04

Societal 11% 3 0,04Scenario 7

Management 14% 4 0,04Scientific 21% 5 0,04

Utility 50% 7 0,07Societal 13% 3 0,04

Scenario 8

Management 17% 4 0,04Scientific 16% 5 0,03

Utility 22% 7 0,03Societal 50% 3 0,17

Scenario 9

Management 13% 4 0,03Scientific 17% 5 0,03

Utility 23% 7 0,03Societal 10% 3 0,03

nter

med

iate

Scenario 10

Management 50% 4 0,13

54/61

54Annex IMetronomeMetronome

Graph 5 Balanced Scenarios

Project#1

Project#2

Project#3

Project#4

Project#5

Project#6

Project#7

Project#8

Project#9

Project#10

Project#11

Project#12

Project#13

Project#14

Project#15

Project#16

Project#17

Balanced (4*25%) 3 16 9 10 7 2 1 15 8 6 17 5 4 13 12 14 11

All-100%(19*0.05) 1 15 10 9 7 3 2 16 8 6 17 5 4 11 14 13 12

0

2

4

6

8

10

12

14

16

18

55/61

55Annex IMetronomeMetronome

Graph 6 Intermediate Scenarios

Project#1

Project#2

Project#3

Project#4

Project#5

Project#6

Project#7

Project#8

Project#9

Project#10

Project#11

Project#12

Project#13

Project#14

Project#15

Project#16

Project#17

Scientific 50% 5 16 9 11 7 2 1 15 8 6 17 3 4 10 12 13 14

Utility 50% 3 16 8 11 4 1 2 13 9 6 17 7 5 14 10 15 12

Societal 50% 1 16 11 8 5 4 3 15 9 7 17 6 2 14 13 12 10

Management 50% 2 8 14 9 10 3 1 15 6 5 17 4 7 11 16 13 12

0

2

4

6

8

10

12

14

16

18

Scientific 50%

Utility 50%

Societal 50%

Management 50%

56/61

MetronomeMetronome

ANNEX II

1. The evaluation matrix

Table 19 below contains the full list of objectives and impact indicators that FP5 and FP6 pro-jects were reviewed against to estimate their potential contribution to sustainable development.The only objectives not detailed within the Table are the Strategic Project objectives, whichhave been withheld to preserve the anonymity of the projects reviewed.

Table 19: Objectives and impact indicators used for the review of FP5 and FP6 pro-jects.

Objectives/impact indicatorsNumber ofprojectsreviewed

FP5 GROWTH• To produce, disseminate and use the knowledge and technologies needed todesign and develop processes and produce high quality, environment- and con-sumer-friendly products which will be competitive on tomorrow's market.• To support the development and implementation of Community policies thatenable competitive and sustainable development.• Development of related services (including transport) which are economic, safeand protective of the environment and quality of life.

6

FP5 EESD To contribute to sustainable development by focusing on key activi-ties crucial for social well-being and economic competitiveness in Europe. 1

FP6 SUSTDEV• To implement a sustainable development model in the short and in the longterm integrating its social, economic and environmental dimensions.• To make a significant contribution to international efforts to mitigate or evento reverse current adverse trends.• To understand and control global change.• To preserve the equilibrium of ecosystems.

1

Wor

k pr

ogra

mm

e ob

ject

ives

FP6 POLICIES• To underpin the formulation and implementation of Community policies byproviding scientific contributions to policies that are targeted precisely on needs.• To ensure timely and effective scientific inputs, covering a wide field of policiesand with the prospect of improved information, exploitation and uptake of re-sults, at national and EU level.• To ensure a coherent research base that reflects the increasing integration ofCommunity policies and of the science that underpins them.• Systematic improvements in the relationship between research and policy at alllevels in the EU.

1

57/61

MetronomeMetronomePr

iorit

y th

e-m

atic

are

aob

ject

ive FP6 IST

• To increase innovation and competiveness in European businesses and indus-try.• To help all European citizens so that they can fully benefit from the develop-ment of the knowledge-based society.

2

FP5 GROWTH Development of fully integrated policy and operational optionsfor an integrated interoperable European rail and road, air and waterbornetransport system on a broadly intermodal basis to ensure the mobility of peopleand goods, while at the same time improving transport efficiency, safety, reli-ability, and reducing congestion and other environmental disbenefits.

5

FP5 GROWTH• Improved fuel efficiency and reduction of emissions - cutting CO2 emissionsand developing and validating zero-emission vehicles.• Improved performance - increasing safety, reliability, maintainability, availabil-ity, operability, energy efficiency and adaptability.

1

FP5 EESD• To help European cities and the associated urban regions to achieve sustain-able economic development and competitiveness.• To help European cities and the associated urban regions to improve man-agement and integrated planning policy.• To help safeguard and improve the quality of life and cultural identity of Euro-pean citizens.

1Key

actio

n ob

ject

ives

FP6 SUSTDEV New technologies and concepts for all surface transport modes(road, rail and waterborne). 1

Prog

ram

me

subd

ivis

ion

obje

ctiv

es FP6 POLICIES The development of tools, indicators and operational parame-ters for assessing sustainable transport and energy systems performance (eco-nomic, environmental and social). 1

Stra

tegi

cob

ject

ives FP6 IST To apply, and extend the scope and efficiency of, IST research to ad-

dress major societal and economic challenges. 2

FP6 IST To develop, test and assess an integrated and global approach to in-telligent road vehicles and aircraft which offers higher safety and value addedservices, where interactions between the person in control, the vehicle and theinformation infrastructure are addressed in an integrated way.

1

Actio

n lin

e ob

ject

ives

FP6 IST• To increase innovation and competiveness in European businesses and indus-try.• To help all European citizens so that they can fully benefit from the develop-ment of the knowledge-based society.

1

Stra

tegi

cpr

ojec

tob

ject

ives Not stated here to preserve anonymity.

11

De-

velo

pm

ent

pol-

Sustainable use of resources, diversified energy for transport, energy efficiency. 11

58/61

MetronomeMetronome

Efficient and harmonised use of pricing measures, e.g. taxation and congestioncharging. 11

High-quality public services for sustainable transport. 11Developing, integrating and managing a more efficient, safer and environmen-tally friendly transport system to provide user-oriented, door to door services forpassengers and freight.

11

Breaking the link between economic growth and increased traffic, and to en-courage more sustainable use of the transport system. 11

Social cohesion, improved safety and security. 11High quality of urban environment. 11Limiting climate change. 11Has the project resulted in the definition of new regulations or regulatory stan-dards? 11

Has the project developed in new policy measures? 11Has the project developed new products/initiatives for the transport sector? 11Have the findings been translated into policy recommendations? 11

1111

Cust

omer

/end

-use

r im

-pa

ct in

dica

tors

Have the project results been disseminated to:Policy makersCivil servantsIndustry 11

Have events been organised for the dissemination of project results? 11

Have the project results been translated into future research needs? 11

Soci

etal

im-

pact

indi

cato

rs

Has society participated in the research? 11

Has the project strengthened or created new research networks? 11Have there been joint publications of project results? 11Have civil servants and/or policy makers participated in the project? 11

Co-o

rdin

atio

nin

dica

tors

Has the project improved networking between researchers and/or public/privateorganisations? 11

2. Realisation of FP5 and FP6 objectives and potential impacts

5.3.1.1.1 Objectives

The set of objectives that were the best met were the Strategic Project objectives. The vastmajority of these were considered to have been ‘fully met,’ with the others (five out of 47) ‘par-tially met’ (see Table 20 below). This is not a surprising finding as the Strategic Project objec-tives are those which are of most direct relevance to the project. The fulfilment of higher levelobjectives, such as Work Programme, Key Action and Programme Subdivision objectives, in-stead indicates whether the project is likely to have broader impacts upon sustainable develop-ment in the EU, which is why it is to be expected that these will not be as fully contributed to asthose that the project was commissioned under. It should also be noted that it is not possibleto evaluate projects based on the extent to which they have contributed to higher level objec-tives. This is largely because the purpose and scope of projects commissioned varies consid-

59/61

MetronomeMetronome

erably and that the magnitude and nature of the impacts and contributions will also vary. Areview against contribution to higher level objectives does, however, provide an indication ofbreadth of contribution.

The low sample size (see Table 20 below) of projects reviewed limits the reliability of findings,but from the projects reviewed it is surprising that the FP5 projects were considered to havecontributed more to Work Programme objectives than the Key Action objectives, and that FP6projects reviewed better met Work Programme objectives than Programme Subdivision objec-tives. This is surprising because the lower level objectives (Key Action and Programme Subdivi-sion) could be considered to be more directly applicable to the projects commissioned.

Table 20: The extent to which FP5 and FP6 projects reviewed contributed to objec-tives and impacts relating to sustainable development.

Extent to which the projectmet/contributed to objectives

Fully/yes Partially IndirectlyNot atall/no

Samplesize

Work Programme 29% 50% 21% 0% 28Strategic 100% - - - 1Priority Thematic Area - 100% - - 2Key Action - - 7% 29% 14Action Line 0% 100% - 0% 2Programme Subdivision 14% 14% 43% 29% 7Strategic project 89% 11% - - 47

Obj

ectiv

es

European Sustainable DevelopmentPolicy

3% 38% 37% 22% 76

Customer/end-user 63% - 1% 36% 67

Societal impact 93% - 3% 3% 30

Impa

ct in

-di

cato

rs

Co-ordination 100% - - - 40

Based upon the reviews conducted the potential contribution and impact of FP5 and FP6 pro-jects is broadly similar, with no meaningful distinction to be made between the two. Perform-ance is broadly comparable, and although there can be seen to be some differentiation betweenthe two FPs this could be owing to the different sample sizes as much as to differing contribu-tions to sustainable development (see Tables 3 and 4 below).

With this caveat in mind it could, however, be said that FP6 projects contributed better to theStrategic Project objectives set with all 14 objectives reviewed considered to have been ‘fullymet’ compared to 28 FP5 Strategic Project objectives being considered to have been ‘fully met’(and an additional five deemed to have been ‘partially met’). As shown in Tables 3 and 4 belowthere is otherwise very little variation between the potential impact and contribution of FP5 andFP6 projects to sustainable development.

60/61

MetronomeMetronome

Table 21: The extent to which FP5 projects reviewed contributed to objectives andimpacts relating to sustainable development.

Extent to which the project met/contributedto objectives

Fully/yes Partially IndirectlyNot atall/no

Work Programme 6 9 2Key Action 2 7 1Strategic Project 28 5

Obj

ectiv

es

European Sustainable DevelopmentPolicy 2 22 12 14

Customer/end-user 25 1 13

Societal 16 1 1

Impa

ctin

dica

-to

rs

Co-ordination 24

Table 22: The extent to which FP6 projects reviewed contributed to objectives andimpacts relating to sustainable development.

Extent to which the project met/contributedto objectives

Fully/yes Partially IndirectlyNot atall/no

Work Programme 2 5 4 -Strategic 1 - - -Priority Thematic Area - 2 - -Key Action - - 1 3Action Line - 2 - -Programme Subdivision 1 1 3 2Strategic Project 14 - - -

Obj

ectiv

es

European Sustainable DevelopmentPolicy - 7 16 3

Customer/end-user 17 - - 11

Societal 12 - - -

Impa

ct in

-di

cato

rs

Co-ordination 16 - - -

On an individual project level there is an inconsistent degree to which the Strategic Project ob-jectives set coincide with wider sustainable development related objectives, both on an EU andan WP level. This is again not a surprising finding as the degree of comparison will almost in-evitably vary based upon the purpose and nature of each project. Only two of the 11 projectsreviewed met their Strategic Project objectives as well as the relevant Key Action (or equiva-lent) objective that they were commissioned under as well as one or more of the relevant Work

61/61

MetronomeMetronome

Programme objectives. A higher number of projects in this category would have indicatedgreater consistency between the different levels of objectives set.

The majority (five) of the projects reviewed only partially contributed to the Key Action orequivalent objective that they were commissioned under, whilst also being considered to havefully contributed to one or more of the Work Programme objectives, despite meeting their Stra-tegic Project objectives. This suggests that there could be considerable discrepancies betweenthe sustainable development components of different levels of objectives set. Contribution towider EU sustainable development objectives was too variable to support any meaningful com-ments in this respect.

5.3.1.1.2 Impact indicators

The potential impact of projects in terms of the customer/end-user, societal and co-ordinationindicators used was consistently positive across the range of projects reviewed. As shown inTables 3 and 4 above there is evidence that either as part of a conscious effort or as part ofstandard FP processes the positive impacts were realised in almost all projects. In the 14 in-stances for FP5 projects and the 11 for FP6 where impacts were not realised this was not theresult of any considerations having been overlooked by the project consortiums, rather that cer-tain indicators, notably whether the projects resulted in the definition of new regulations, stan-dards or policy measures, were not applicable to the programmes of work being undertaken.