Post on 04-Jan-2016
Strategic Evaluation of Research
–Viewpoint of Outcome in the Evaluation of Research Units
in AIST–M. Daté, H. Tokunaga, S. Nakatsu, S. Ito,
H. Matsuhata, H. Ogi, A. Omori, S. Suto, M. Tajima, T. Tanaka, K. Urabe, O. Nakamura*, O. Nakamura, S. Kosaka, M. Takagi Sawada and N. Kobayashi
Natl. Inst. Adv. Ind. Sci. Technol. (AIST), JAPAN*Min. Econ. Trade Ind. (METI), JAPAN
1/15
Outline of AISTNew AIST (National Institute of Advanced Industrial Science and Technology) was established in 2001 as an Independent Administrative Institute.
ResearchersAdministrative StaffsTotal Employees
2,505704
3,209 As of April 2006
2/15
Mission of AISTBasic Principle is “Realization of a Sustainable Society”.
3/15
Research Fields in AIST
4/15
Organization of AIST
As of August 2006
5/15
Collaboration Dept
Auditor
President
Research Centers
Research Institutes
Research Initiatives
Senior Vice-PresidentVice-President
Emeritus Advisor
AIST Advisory Board
Research Coordinator
Emeritus Councilor
AIST Fellow
Research Units
adm
inis
trat
ive
un
its
Intellectual Property Dept
Technology Information Dept
International Affairs Dept
General Administration Dept
Human Resources Dept
Financial Affairs Dept
Research Facilities Dept
Intl Patent Organism Depositary
AIST Innov Center for Start-ups
Geo-information Center
Metrology Management Center
Tsukuba Adv Comp Center (TACC)
Planning HQs
General Administration HQs
Evaluation Dept
Safety & Environ Protection Dept
Public Relations Dept
Legal Office
Info Disclosure & Personal Info Protection Promotion Office
Gender Equality Office
Audit Office
Research Units of AIST
Research Centers (29)
Priority in investing research resources
Operated by top-down management
Pioneering and strategic projects within a certain period (<7 years)
Research Institutes (21)
Proposal of research subjects in a bottom-up manner
Mid- and long-term researches
Research Initiatives (7)
Response to sudden administrative needs
Aiming for new research centers or fields within a short period (<3 years)
As of August 2006
6/15
Evaluation of Research Units
Purposes of evaluation
2) Reference to the top management of AIST
3) Accountability to the Japanese Nation
1) Activation and efficient management of Research Units
1st Mid-Term (FY2001-2004) 2nd Mid-Term (FY2005-2009)
Evaluation based on Outputs and global level of research
Evaluation from the viewpointof Outcome
Viewpoint of evaluation
7/15
Definition of Terms in AISTOutputs:
Direct results in R&D (e.g., papers, patents, etc.)
・・ BudgetBudget・・ Human Human resourceresource・・ EquipmentEquipment
・・ Academic Academic paperpaper・・ PatentPatent・・ Geologic MapGeologic Map・・ StandardStandard・・ SoftwareSoftware・・ Process Develop.Process Develop.・・ DatabaseDatabase
・・ Popularization Popularization of productsof products・・ New research New research fieldfield・・ ServiceService・・ Saving energy Saving energy ・・ Risk Risk managementmanagement・・ DisasterDisaster preventionprevention
INPUT IMPACTOUTPUT OUTCOME
・・ GDP growthGDP growth・・ New employmentNew employment・・ InternationalInternational competition abilitcompetition abilityy・・ High QoLHigh QoL・・ Environmental Environmental protectionprotection・・ Security & safety Security & safety societysociety
Outcomes:Social and economical effects caused by outputs (e.g., products, standards, novel research fields, etc.)
8/15
Evaluation System in AIST
Outputs in the past Realized Outcomes
Expected Outcomesin the future
1) Roadmaps2) Outputs3) Management
PAST PRESENT FUTURE
Subjects of evaluation
2) Evaluation of Outputs toward Outcomes
3) Evaluation of Management toward Outcomes
1) Evaluation of Roadmaps toward Outcomes
9/15
Evaluation IndexesEvaluation of Roadmaps
Outcomes, Milestones, Technical Elements, Benchmarks
Evaluation of Outputs
Papers, Patents, Standards, Academic Activities, Press Releases, Prizes, Prototypical Products, etc.
Evaluation of Management
Research Strategy
Other Management
1) Full Research, 2) Innovations, 3) Human Resource Development
1) Overall Management, 2) Budget Management, 3) Risk Management
10/15
Reviewers in EvaluationExternal Reviewers (5-7 for each Research Unit)
e.g., academia, industry, journalism, etc.
Specialists and intellectuals outside of AIST
Internal Reviewers (3 for each Research Unit)
Principal Reviewers of AIST, etc.
Main Role: Evaluation of Roadmaps and Outputs
Main Role: Evaluation of Management
(Roadmaps & Outputs) : Management = 7 : 3
Allocation of grades
11/15
Evaluation Procedures
Resource, ReorganizationResource, Reorganization
Evaluation Committee
Evaluation Committee
1) Roadmaps2) Outputs3) Management
Evaluation DepartmentEvaluation Department
Activity Report
Presentation
Activity Report
PresentationComments on Evaluation
Claim of Reconsideration
Comments on Evaluation
Claim of Reconsideration
Grades Comments
Grades Comments
CorrectionCorrection
External ReviewersInternal Reviewers
External ReviewersInternal Reviewers
Research UnitResearch Unit
12/15
Evaluation Committeeof Ind. Admin. Inst.Evaluation Committeeof Ind. Admin. Inst.
PresidentPresidentJapanese NationJapanese Nation
Research Strategy of AISTResearch Strategy of AIST
Overall Evaluation Overall Evaluation
Survey of the Unit Directors (1)
Was the evaluation of this year useful?Was the evaluation of this year useful?
Was the evaluation worth all the efforts?Was the evaluation worth all the efforts?
FY2005
FY2005
FY
FY
13/15
Survey of the Unit Directors (2)
Was the viewpoint of outcomes useful?Was the viewpoint of outcomes useful?
Was the evaluation useful in improving management?Was the evaluation useful in improving management?
FY2005
FY2005
FY
FY
14/15
Comments of External ReviewersCATEGOLY PERCENTAGE
Affirmative comments on the evaluation system
20
Comments on the evaluation of roadmaps
14
Misunderstanding, insufficient explanation
14
Insufficient time or activity reports
12
Proposal to improve the evaluation system
10
Request for various evaluation indexes 9
Request for a longer interval of evaluation
8
Request to reduce the load of evaluation
6
Other comments 9
FY 2006
15/15