Follow-up Evaluation to Advance R&D Policy - Scenario … · History of evaluation in ... (space,...
-
Upload
duongthuan -
Category
Documents
-
view
214 -
download
0
Transcript of Follow-up Evaluation to Advance R&D Policy - Scenario … · History of evaluation in ... (space,...
Follow-up Evaluation to Advance R&D Policy
- Scenario from long term vision -
Yasukuni OKUBO, [email protected]
Director for Technology Evaluation,Technology Evaluation and Research Division
Ministry of Economy, Trade and Industry
Joint TIR/RIHR Workshop on Enhancing Research Performance through Evaluation and Priority
Sept. 15-16, 2008
Contents
Introduction of evaluation at METI and others
2
Introduction of Follow-up Evaluation at METI –Lithium Battery-Introduction of Follow-up Evaluation at NEDOConclusions
01Jan Restructuring of Central Government 22 Ministries & Agencies -> Cabinet Office+12 Ministries
96 1st Science and Technology Basic Plan97 General Guideline for Evaluation of Research and Development
History of evaluation inJapanese public institutions
Introduction of evaluation at METI and others
01Mar 2nd Science and Technology Basic Plan01Apr Independent Administrative Institutions (IAI)
to separate Operational Sectors from Central Government01Jun Government Policy Evaluation Act01Nov Revision of General Guideline for Evaluation of Research and Development02Jan Establishment of The Council for Science and Technology Policy (CSTP)02Apr METI Guideline for Technology Evaluation02Apr METI Standard Items and Criteria of Evaluation
05Mar Revision of General Guideline for Evaluation of Research and Development06Mar 3rd Science and Technology Basic Plan
Grant & contract ResearchCreation of technology
seedsEstablishment of new technology for next
generation
Tax Reduction
Tax credit on increased
R&D expensesNEDO
METI (17% of total S&T budget)MEXT(64%)
METI’s Role in S&T AdministrationMETI’s Role in S&T Administration
N ti l P j t
Subsidy etc.
503 BillionJY
217BillionJY
Subsidy etc.
Introduction of evaluation at METI and others
4
AISTNational Institute of
Advanced Industrial S&T
Industries
Reform of national innovation system(industry - university cooperation, national project reform, IPRs, etc,)
UniversitiesCooperation
Economy
TechnologyTransfer
National Projects(space, nuclear)
TLO
60BillionJYNew Energy and Industrial Technology Development Organization
Mission: Securing sustainable economic growth and stable growth of international economy, through enhancement of Japanese competitiveness, creation of new market,
and improvement of socioeconomic infrastructure.
Policy and program system at METI
6 policies/34 programs /Over 120 projects
1. Economic and Industrial Policy2. Foreign Economic Policy3. Policy for Manufacturing I d t i I f ti &
Programs Projects
Programs Projects
P
Introduction of evaluation at METI and others
6. Nuclear and Industrial Safety Policy
Industries, Information & Communication, and Services4. Policy for Small and Medium-sized Enterprises & Regional Economy and Industry5. Energy & Environmental Policy
Programs Projects
Programs Projects
Programs Projects
Programs Projects
METI Time Table and Methodology0th Year 1st Year ・ ・ ・ ・ ・ ・ Next Year After 5 Years
Ex-ante Interim Ex-post Follow-up
Start End
Sub-committee
Panel meeting WG
SelfExternalM
onito
ring
Mon
itorin
g
Sub-committee
Panel meeting
ExternalPanel meeting
External
Introduction of evaluation at METI and others
Hearing ● ● ●
Logic Model ○ ○ ○ ○
Indicator ● ● ● ● ● ●
Score ● ●
Peer Review ● ● ●
Expert Review ● ● ●
Quantitative(Bibliometrics)
○ ○ ●
Interview/Questionnare
○ ● ●
Range of Evaluation at R & D Projects at METI
研究開発プロジェクト
インプット資金
アクション
研究
カス
直接
ア
プロ
ジ 施策成果
プロジェクトの範囲施策の範囲
アウトカム
インパクト
研究開発プロジェクト
インプット資金
アクション
研究
カス
直接
ア
プロ
ジ 施成果
プロジェクトの範囲施策の範囲
アウトカム
インパクト1 2 3
研究開発プロジェクト
インプット資金
アクション
研究
カス
直接
ア
プロ
ジ 施策成果
プロジェクトの範囲施策の範囲
アウトカム
インパクト
R & D Project
mer
Range of ProjectRange of Policy
Outcome
Impact
of P
olic
y
urpo
se
tcom
e
put M
easu
res)
nage
men
t)
tion
Org
aniz
atio
n)
Evaluation of Impacts to industry and Cost Benefits
Outputs Evaluation
Introduction of evaluation at METI and others
7
資金期間組織
研究テーマ
事業テーマ
スタ
マー
アウ
トカ
ム
ジェ
クト
目標
策目
標(成果指標)
ここまではしっかり描く この部分は簡単に描いてみる
資金期間組織
研究テーマ
事業テーマ
スタ
マー
アウ
トカ
ム
ジェ
クト
目標
策目
標(成果指標)
ここまではしっかり描く この部分は簡単に描いてみる
4 5 6
資金期間組織
研究テーマ
事業テーマ
スタ
マー
アウ
トカ
ム
ジェ
クト
目標
策目
標(成果指標)
ここまではしっかり描く この部分は簡単に描いてみる
Cus
tom
Proposal for better strategy
Obj
ectiv
es
Proj
ect P
u
Dire
ct O
ut
Out
puts
(Out
p
Actio
n (R
&D M
a
Inpu
ts (S
alar
y D
urat
In the innovation policy, outcomes are significant.It is not easy to evaluate the outcomes, butWe have been challenging to evaluate outcomes in project evaluation.
Range of METI, AIST & NEDO in terms of R&D
研究開発プロジェクト
インプット資金
アクション
研究
カス
直接
アウ
プロ
ジェ
施策成果
プロジェクトの範囲施策の範囲
アウトカム
インパクト
研究開発プロジェクト
インプット資金
アクション
研究
カス
直接
アウ
プロ
ジェ
施策成果
プロジェクトの範囲施策の範囲
アウトカム
インパクト1 2 3
研究開発プロジェクト
インプット資金
アクション
研究
カス
直接
アウ
プロ
ジェ
施策成果
プロジェクトの範囲施策の範囲
アウトカム
インパクトMETI
Basic Research
NEDO
Applied Research
Executing Body
Regulation, Tax, etc.
Purp
ose
Range of Policy
ImpactOutcome
es o
f Pol
icy
Introduction of evaluation at METI and others
Direct Outcome
Project Manager
8
資金期間組織
研究テーマ
事業テーマ
タマ
ー
ウト
カム
ェク
ト目
標
策目
標(成果指標)
ここまではしっかり描く この部分は簡単に描いてみる
資金期間組織
研究テーマ
事業テーマ
タマ
ー
ウト
カム
ェク
ト目
標
策目
標(成果指標)
ここまではしっかり描く この部分は簡単に描いてみる
4 5 6
資金期間組織
研究テーマ
事業テーマ
タマ
ー
ウト
カム
ェク
ト目
標
策目
標(成果指標)
ここまではしっかり描く この部分は簡単に描いてみる
AIST
(Executing Body)Customer
Responsible Range of National R&DPr
ojec
t P
Range of Project
Obj
ectiv
e
NEDO: New Energy and Industrial Technology Development Organization
AIST: National Institute of Advanced Industrial Science and Technology
TLO, etc
Introduction of Follow-up Evaluation at METI
NEDO’s Follow-up and METI’s Follow-up
METI
Executing Body
pose
Impact
fPol
icy
Introduction of Follow-up Evaluation at METI
Range of METI’s Follow-upRegulation, Tax, etc.
OutcomeRange of NEDO’s Follow-up
pose
9
NEDO
AIST
(Executing Body)TLO, etc
Proj
ect P
urp
Range of Policy
Obj
ectiv
es o
f
CustomerRange of Project
Proj
ect P
urp
Evaluation Items for Follow-up Evaluation at METI
Ⅰ.ImpactsⅠ-1. Technology ImpactⅠ-2. R&D Performance ImprovementⅠ-3. Economic BenefitⅠ-4 Social Life Improvement
Extent of progress
Introduction of Follow-up Evaluation at METI
E id
10
Ⅰ 4. Social Life ImprovementⅠ-5. Feedback to Policy
Ⅱ.Evaluation from current positionⅡ-1. Justification of projectⅡ-2. TargetⅡ-3. PlanningⅡ-4. Justification of ex-post evaluation based on results of Ⅱ-1~Ⅱ-3Ⅱ-5. Follow-up after project
Accelerating or Impeding Elements to Outcomes
Evidences
Role of Follow-up EvaluationIntroduction of Follow-up Evaluation at METI
Few evidences remain
Missing linkages for negative elements
11
295,624
248,207250,966
305,548
293,751289,148
30,00060,000
400,000
600,000(百万円)
自動車用
小型民生用 将来的市場ポテンシャル大
30万台/年として2020年600億円
HEV 15万台/年として2010年300億
リチウム二次電池市場将来予測Perspective of Lithium Battery MarketMillion Yen
High Potential in Market
0.3 M cars/ya -> 60 B.Yen in 2020
0.15 M cars/ya -> 30 B.Yen in 2015
HEV
CarsSmall-scale Private Use
Strong point:Higher energy density storage than Lead-acid battery.
Introduction of Follow-up Evaluation at METI–Lithium Battery-
12High Performance Battery Project
Lithium Stationary Battery Project200720062002
2001
300,000 300,000
0
200,000
2000 2001 2002 2003 2004 2005 2010 2020 (年)
小型民生用は3000億円/年レベルで推移300 B.Yen for Small-scale Private Use
(Year)
2011Lithium Battery for FC Project
1992 2006 2010Decentralized Battery Project
①Hybrid Car (Nissan 2000) Shin-Kobe Electric Machinery made Battery
Application to Railway Car
Application to Large Scale cars
Application to Bike
○Down-sizing and weight
⑩Li Battery mounted Cart (shin-Kobe Electric Machinery 2004) Shin-Kobe Electric Machinery made Battery
Application to Care Goods of Wheelchair, etc.○D i i
Application to Gulf Cart○High Density Output○Module Control
⑪Li Battery Test Module for Satellite (SERVIS-1 Satellite 2003) Mitsubishi Electric ・GS Yuasa Cooperation
Application to Satellite
○Thermal Simulation○Down-sizing and weight reduction by Improvement of energy density
③EV(Nissan 2000) Shin-
②Addling Stop Car (Toyota 2005) Mitsubishi Chemical Made Battery material
○High Density Output○Module Control○Design for Circuit○Material Technology
HEV,PEV etc. for low cost & less pollution
Introduction of Follow-up Evaluation at METI–Lithium Battery-
Lithium Battery for Hybrid
⑨LI Battery mounted Wheelchair (Yamaha 2005) Hitachi Vehicle Battery
Car
④Hybrid Bus(MitsubishiFuso Truck Bus 2004) GS Yuasa made Battery
⑧Li Battery mounted Diesel Car
(East Japan Railway Company 2007 Test Car) Shin-Kobe Electric Machinery made Battery
⑦Motor Scooter(Yamaha 2003) Hitachi Vehicle Battery
⑤Hybrid Truck (Isuzu 2005) Hitachi Vehicle Battery
⑥Motor Assist Bike(Yamaha Sanyo made Battery
○High Density Output
○Large-scale cell ○Large-scale module
and weight reduction by Improvement of energy density
y y○Down-sizing and weight reduction by Improvement of energy density
2000) Shinkobe Electric Machinery made Battery
Output Measures
1MW class safe-first battery storage
Energy density: 70Wh/kg
Lithium Stationary Battery Project 2006 - 2010
Targets :
40000 yen/kWh combined with the wind power generation.
Introduction of Follow-up Evaluation at METI–Lithium Battery-
Decline of demand for electricity, cost down, and conventional regulation impeded to create new demand of battery storage system for family.
Life time: 10 yearsTotal system efficiency: >80%Cost: 40000 yen/kWh
gStationary type has a potential to grow for stabilizing power from wind power generation.
Introduction of Follow-up Evaluation at NEDO
◆To ensure accountability to Japanese taxpayers~by monitoring post-project activities of project participants with regard to the
practical application of R&D achievements.
Purposes of Follow-up monitoring at NEDO
Introduction of Follow-up Evaluation at NEDO
15
Evaluation at NEDO◆ To provide feedback for improving NEDO’s R&D
management~by reviewing past post-project evaluations and post-project activities.
◆ To contribute to the planning of future R&D strategies~by interpreting technology trends in post-project research activities.
Prospect for practical use
What is a Follow-up Chart?A follow-up chart is a tool which chronologically identifies significant events in project management.The main objective is to show up “commercialization” or “discontinuation” scenario in an executing body. Monitoring projects every year, NEDO draws the chart based on interviews to a company concerned. An interviewee is a person who understands the whole project and its evaluation. The horizontal axis represents time while the vertical axis represents prospect for practical use of technology. The coordinates along the vertical axis are based on the opinions of interviewees.
Follow up Chart
Introduction of Follow-up Evaluation at NEDO
Prospect for practical use
Time (years)
Planning Project Post-project
Turning points and their influences/effects on management for practical use
Large
Mid
Small Present
Follow-up Chart
Type A Type B
Achieved Commercialization
Until now, NEDO conducted interviews to 75 companies. Out of them, 46 companies achieved commercialization and 23 stopped R&D to achieve commercialization.The analysis of the interviews categorized them into four different types.
Too optimistic to solve technical problems
Four Types of Follow-up Charts
al u
se
use
Accomplish delineation of technical problems at time of planning.
Introduction of Follow-up Evaluation at NEDO
33/46 13/46
technical problems.R&D breakthrough during latter half of project.
Large
Pros
pect
for p
ract
ica
Mid
Small
Large
Mid
Small
Project TimeProject Time
Pros
pect
for p
ract
ical
Steadily overcome technical problems.R&D based on clear vision for social/ user needs. .
Discontinued R&D after Project
Type DType C
Too optimistic to solve technical problems.No R&D breakthrough during latter half of project
Steadily overcome technical problems.No needs and no cost-competitiveness just after
Four Types of Follow-up Charts Introduction of Follow-up Evaluation at NEDO
8/2315/23
during latter half of project.competitiveness just after the project..
Large
Mid
Small
Large
Mid
Small
Project TimeProject Time
Pros
pect
for p
ract
ical
use
Pros
pect
for p
ract
ical
use
Advantages of Follow-up Monitoring and Evaluation
(2)Dissemination and promotionConducting the follow-up monitoring, NEDO sometimes finds that technology developed in the death-valley. If necessary, NEDO would
id fi i l i i d b l
(1) Share knowledgeThe management lessons obtained from follow-up monitoring are accumulated in the “NEDO R&D Project Management Guideline” accessible through the NEDO’s intranet system.
Introduction of Follow-up Evaluation at NEDO
provide financial assistance in order to remove obstacles.
(3)Policy proposals to the governmentNEDO submits policy proposals including results from follow-up monitoring. Reform of laws and the national innovation system is an example.(4)Opinion exchange with industry
The lessons derived from follow-up monitoring results are shared with the companies participating in NEDO projects. Discussions related to policies required to promote innovation are also carried out.
ConclusionsLegitimacy of policy interventions and relevance of evaluations
To monitor diffusion of products Role of ex-post evaluations
To advance strategy on R&D
20
gyBetter link evaluation results with research budgeting mechanisms and decisions
To identify undiscovered R&D program, funding, regulation, etc.
“Policy mix” for research and innovation policiesTo overview and monitor the whole from outputs to social effects