Workshop Evaluation, Berlin, 26/27 September 2005

WISSENSCHAFTSRAT

International Workshop on theEvaluation of Publicly Funded Research

Berlin, 26/27 September 2005

Summative evaluations and their effects on institutional strategy and performance.

Experiences from the German Wissenschaftsrat‘s evaluations of Blue List institutions

Professor Dr. Reinhard HüttlChairman of the Scientific Commission

of the German Wissenschaftsrat

WISSENSCHAFTSRAT

Workshop Evaluation, Berlin, 26/27 September 2005

WISSENSCHAFTSRAT

Scientific Scientific membersmembers

Scientific Scientific membersmembers

Federal and state Federal and state governmentsgovernments

Federal and state Federal and state governmentsgovernments

The German Wissenschaftsrat

Committees and working groupsCommittees and working groups(complemented by external experts) (complemented by external experts)

prepare recommendationsprepare recommendations

Committees and working groupsCommittees and working groups(complemented by external experts) (complemented by external experts)

prepare recommendationsprepare recommendations

send members

Workshop Evaluation, Berlin, 26/27 September 2005

WISSENSCHAFTSRAT

Level 2Comparative assessment

Level 1Scientific assessment

Evaluation procedure

Working group:evaluation report

Working group:evaluation report

Evaluation committee/Wissenschaftsrat:

Science policy evaluation statement

Evaluation committee/Wissenschaftsrat:

Science policy evaluation statement

Workshop Evaluation, Berlin, 26/27 September 2005

WISSENSCHAFTSRAT

Experience in institutional evaluation:

• Ex-ante evaluations concerning large scale investment in scientific instruments• Cross sectional evaluations of research fields• System evaluations of research organisations• Structural evaluations of specific disciplines• Evaluations of single research institutes• Research rating• Comparative evaluations of institutional strategies

Workshop Evaluation, Berlin, 26/27 September 2005

WISSENSCHAFTSRAT

„Blue list“ institutes:

• Approximately 80 non-university institutes• Jointly funded by Federal Government and state governments• Missions include research, service and policy advice• Evaluation of all institutes completed in 2001

Workshop Evaluation, Berlin, 26/27 September 2005

WISSENSCHAFTSRAT

Criteria for institutional evaluation:

• Core criterium: quality of scientific work• Peer review-based methodology• Flexible adjustment of general criteria

Workshop Evaluation, Berlin, 26/27 September 2005

WISSENSCHAFTSRAT

Effects of Blue List evaluation

• gains in reputation• shared funding stopped in five cases, nine new affiliations• improved organisation and research strategy

Workshop Evaluation, Berlin, 26/27 September 2005

WISSENSCHAFTSRAT

Implementation of results

• evaluation must have consequences• Wissenschaftsrat keeps track of implementation• shared interest of science and science policy not to damage the instrument of evaluation

Workshop Evaluation, Berlin, 26/27 September 2005

WISSENSCHAFTSRAT

Evaluation of institutes of scientific policy advice:

• Policy advice needs firm scientific basis

• Evaluation of governmental research institutes currently under way

Workshop Evaluation, Berlin, 26/27 September 2005

WISSENSCHAFTSRAT

Further developments

• Research rating: pilot study on chemistry and sociology

• Excellence initiative: institutional strategies to develop top-level university research