1

Introduction of Research and Development Project Evaluation System at NEDO

Momoko OKADANew Energy and Industrial Technology Dev

elopment Organization(NEDO)

28 October 2005CES/AEA Conference in the Sheraton Centre, Toronto

2

-CONTENTS-

• ABOUT NEDO

• R&D EVALUATION SYSTEM AT NEDO -INTERMEDIATE & EX-POST EVALUATIONS -FOLLOW-UP MONITORING& EVALUATIONS

• CLOSING

3

-ABOUT NEDO-

• Establishment -Established in October 1980 as a semi-governmental organization under

the supervision of METI. - Original activities :　　 ・ national R&D projects on new energy

-Current activities : ・ Support of industrial technology R&D (in 1988), ・ Support of environmental technology R&D (in 1990), and ・ Introduction and promotion of new energy and energy conservation technology( in1993).

-NEDO became an Incorporated Administrative Agency as of October 2003 in the process of recent government reform.

4

• Mission -NEDO’s current missions are:　 ・ Promote new energy and energy conservation technologies through

R&D and subsidy programs,

・ Conduct and support industrial technology R&D, and

・ Enhance international cooperation concerning energy and other

technologies.

-ABOUT NEDO-

5

• Organization

Staff : approximately 1,000 Budget : US$2.1 billion (FY2005)

-ABOUT NEDO-

Machinery System Technology Development Dept.

Environment Technology Development Dept.

Fuel Cells & Hydrogen Technology Development Dept.

New Energy Technology Development Dept.

Energy Conservation Technology Development Dept.

Coal Projects Dept., Coal Mine Subsidence Dept.

Biotechnology & Medical Technology Development Dept.

Nanotechnology & Materials Technology Development Dept.

Electronic & Information Technology Development Dept.

International Projects Dept.

Energy and Environment Policy Dept.

R&D Promotion Dept.

R&D Project Evaluation Dept.R&D Project Evaluation Dept.

Policy Planning and Coordination Dept

General Management Depts. General Affairs; Accounting; Assets Management; Inspection and Operational Management; and Information and Systems.

Chairman, President & Executive Directors AuditorHeadquarters

6

• R&D support -R&D support is NEDO’s core activity.

・ 58% of the total budget (US$1.2 billion) is allocated for R&D activities in FY 2005.

・ Number of NEDO’s R&D projects in FY2005 is 174.

-ABOUT NEDO-

- Electronics and information - Environment- Nanotechnology and materials - Fuel cells and hydrogen- Biotechnology and medical technology - New energy- Machinery systems - Energy conservation

NEDO’s R&D Technology Areas

7

• Challenge for NEDO -Need more effective and efficient operation through:

・ Improved services, and

・ Flexible resource management

-ABOUT NEDO-

8

• R&D Project Management Cycle

PLANPLAN

DODOSEESEE

R&D Project Evaluation Dept.

Board ofExecutive Directors

Board ofExecutive Directors Technology

Development Depts.

Policy Planning and Coordination Dept.

-R&D EVALUATION AT NEDO-

9

• Purposes of R&D Project Evaluations as an important part of the R&D project management cycle.

-To reflect economic and social needs on R&D activities,

-To implement R&D projects more effectively and efficiently, and

-To improve project management system

-R&D EVALUATION AT NEDO-

10

Transparent -Public disclosure of evaluation results, methods, and reflectionsExplicit -Encourage discussionsEffective -Adopt an evaluation method for self- improvement of managementNeutral -External evaluators Efficient -Simple evaluation procedures

• 5 Principles of Evaluation at NEDO

-R&D EVALUATION AT NEDO-

11

• R&D Project Evaluations

Post-project activityProject implementation

Ex-anteevaluation

(2004-)

At R&D theme selection

Ex-anteevaluation

(2004-)

At R&D theme selection

Intermediateevaluation

(2001-)

In the middle of project term

Intermediateevaluation

(2001-)

In the middle of project term

Ex-postevaluation

(2001-)

1 year followingproject completion

Ex-postevaluation

(2001-)

1 year followingproject completion

▼ ▼ ▼ ▼

Follow-upmonitoring &

evaluation(2004-)

For 5 years afterproject completion

Follow-upmonitoring &

evaluation(2004-)

For 5 years afterproject completion

-R&D EVALUATION AT NEDO-

12

• Intermediate & Ex-post Evaluations

-In all procedures, 5 principles are followed

-Each project is assessed and evaluated from the following areas :

・ purpose and strategy,

・ project management,

・ R&D achievements, and

・ prospect for practical applications and other impacts

-R&D EVALUATION AT NEDO-

13

• Intermediate & Ex-post Evaluations - R&D Evaluation Methodology (Rating Method)

1. Each evaluator grades (A, B, C or D) for each evaluation item. A: Excellent, B: Good, C: Acceptable, D: Not acceptable2. An average of rating is used as an indicator. A=3, B=2, C=1, D=0

Items for evaluation average ratings

1. Purpose and strategy 2.3 A A B B B B

2. Project management 1.8 A B C B C B

3. R&D achievement 1.5 B C C C B B

4. Prospect for practical applications and other impacts 1.2 B B D C C C

-R&D EVALUATION AT NEDO-

【 example】

14

-R&D EVALUATION AT NEDO-

1.0

1.5

2.0

2.5

3.0 12

21

225,10,28

29

14

26

179

18

62

3,4

1

7

8,30

23

16

13,24

15

27

9

11

20

25

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.0 0.5 1.0 1.5 2.0 2.5 3.0

R&D achievement (X)Pro

sp

ect

for

pra

cti

cal ap

plicati

on

s （Ｙ

）X+Y=4

X+Y=3Good

Excellent

Score Results for Ex-post Evaluation in FY2004

15

• Intermediate & Ex-post Evaluations

Public (citizens)

NEDO

Chairman

Releasing of evaluation results to the public

Deliberation and approval of the evaluation report (draft)

Reflection of evaluation results in projects, etc. Promoting

DepartmentEvaluation report

Preparation of evaluation report (draft)

Promotion Department

Project Participants

SubcommitteeA

SubcommitteeC

SubcommitteeB

SubcommitteeD

Evaluation Secretariat

Explanation of the project

(4-9 persons）

(13 persons）R&D Project Evaluation Committee

-R&D EVALUATION AT NEDO-

16

-R&D EVALUATION AT NEDO-• Intermediate & Ex-post Evaluations

-In response to evaluation results:• Lessons learned from ex-post evaluations may be reflected

upon other on-going or new projects.

• Intermediate evaluations may result in modification or termination of projects.

Responses to Intermediate Evaluation Results

2001 2002 2003 2004

Accelerated - - 2 13

Continued as planned 4 2 12 12

Modified 16 6 15 6

Terminated 2 5 2 2

total 22 13 29* 29*

*some items are overlapping: all items added up do not match the total

(start)

17

-R&D EVALUATION AT NEDO-

• Follow-up Monitoring & Evaluations (since 2004)

-Objectives of this new initiative are:

　・ To understand in what ways specifically the outcome of R&D provides benefits to the public

・ To reflect the outcome of R&D in improvement of the management and planning function at NEDO

18

• Follow-up Monitoring & Evaluations

-R&D EVALUATION AT NEDO-

Tracked with the Follow-up Monitoring

Post-Project Activities

NEDO’s R&D Project(3-5 years)

R&DPractical

ApplicationsCommercialization

Post-Project Activities of Private Companies

19

• Follow-up Monitoring & Evaluations-NEDO initiatives as of October 2005: Subjects: 56 R&D projects , 642 entities

・ These projects subject to follow-up monitoring & evaluations have received ex-post evaluation in FY2002-2003.

・ These entities include private companies, universities, research institutes, and public institutions.

-R&D EVALUATION AT NEDO-

20

• Follow-up Monitoring & Evaluations• Findings:

-64% of the above entities are currently conducting post-project activities.

-22% of the entities have reached a stage of practical applications or commercialization of R&D achievements.

*Detailed information is being collected from 51 entities.• Utilization: -Make suggestions for improvement -Document key points to R&D management as a guideline

-R&D EVALUATION AT NEDO-

21

・ Evaluations will play an increasingly important role in: -Improving performance of NEDO as an Incorporated

Administrative Agency, and -Facilitating effective and efficient implementation of R&D project managements.

CLOSING

22

CLOSING

• NEDO intends to

・ Review the current evaluation system

・ Make appropriate modifications

・ Propose new initiatives

23

Thank you for your kind attention.

To be continued…