Moscow January 21, 2015 An overview of STI statistics and indicators Giorgio Sirilli ISCrES - CNR.
-
Upload
avis-morris -
Category
Documents
-
view
221 -
download
1
Transcript of Moscow January 21, 2015 An overview of STI statistics and indicators Giorgio Sirilli ISCrES - CNR.
MoscowJanuary 21, 2015
An overview of STI statistics and indicators
Giorgio SirilliISCrES - CNR
2
Outline of the presentation
STI indicatorsSTI policyThe actorsModels of innovationManualsUses and abuses of indicatorsConcluding remarks
3
Science, technology and innovation
Science is the reasoned investigation or study of phenomena, aimed at discovering enduring principles among elements of the phenomenal world by employing formal techniques such as the scientific method
Technology is the collection of tools, including machinery, modifications, arrangements and procedures used by humans
Innovation is the process practical application of knowledge translating an idea or invention into a good or service that creates value or for which customers will pay
4
Statistic: A numerical fact or datum, i.e. one computed from a sample.
Statistical data: Data from a survey or administrative source used to produce statistics.
Statistical indicator: A statistic, or combinations of statistics, providing information on some aspect of the state of a system or of its change over time. (For example, gross domestic product (GDP) provides information on the level of value added in the economy, and its change overtime is an indicator of the economic state of the nation.)
Indicators
5
Indicators are a technology, a product, which- governs behaviour- is modified by users (outside of the producer community)- develops in response to user needs
Data sources– Surveys, administrative data, private files, case studies– Data collection is informed by manualsData populate statistics which can be indicatorsDecisions are taken on the basis of indicators
Indicators
6
Science and technology indicators
S&T indicators are defined as “a series of data which measures and reflects the science and technology endeavor of a country, demonstrates its strengths and weaknesses and follows its changing character notably with the aim of providing early warning of events and trends which might impair its capability to meet the country’s needs”.
Indicators can help “to shape lines of argument and policy reasoning. They can serve as checks, they are only part of what is needed”.
(OECD, 1976)
7
“If you can not measure it, you can not improve it.”
Lord Kelvin
Lord Kelvin
8
“If you can not measure it, don’t talk about it.”
Ariolf Grupp
Hariolf Grupp
9
The evolution of STI indicators
The first attempt to measure S&T in 1957Frascati Manual (1963)The Frascati manual “family”A continous process of broadening and deepening: from
macro to micro, from public to privateThe role of international organisationsThe dialogue between producers and users
10
Science and technology policy
A history which starts after WW2
1940s 2010s
science technology innovation competitiveness social needs
jobs
11
Report “Science the Endless Frontier” 1945 (Vannevar Bush)
Science and technology policy
12
“Science the Endless Frontier”
Concerns: Military security; Health
Solution: Science policy
“The Government is particularly fitted to perform certain functions, such as the coordination and support of broad programs on problems of great national importance”
“Scientific progress on a broad front results from the play of free intellects, working on subjects of their own choice, in the manner dictated by their curiosity for exploration of the unknown. Freedom of inquiry must be preserved under any plan for Government support of science”
13
“Publicly and privately supported colleges and universities and the endowed research institutes must furnish both the new scientific knowledge and the trained research workers. It is chiefly in these institutions that scientists may work in an atmosphere which is relatively free from the adverse pressure of convention, prejudice, or commercial necessity. At their best they provide the scientific worker with a strong sense of solidarity and security, as well as a substantial degree of personal intellectual freedom.”
“Industry is generally inhibited by preconceived goals, by its own clearly defined standards, and by the constant pressure of commercial necessity. Satisfactory progress in basic science seldom occurs under conditions prevailing in the normal industry laboratory.”
“Science the Endless Frontier”
14
The triangle in STI policy making
Policy makers design the future
Analysts interpret today
Data producers measure the past
15
Luigi EinaudiItalian President
“Conoscere per deliberare”
Know first and then sanction
Policy makers
16
Policy makers
President Roosevelt Vannevar Bush
Report “Science the Endless Frontier” 1945
17
Policy makers
“Why shoud we pay the researchers if we make the best shoes in the world?”Silvio Berlusconi
“Culture does not provide food”(Con la cultura non si mangia)Giulio Tremonti
18
After WWII the main issue in STI policy was the social responsibility of science. Now the attention is placed on the social return of investment on STI and, in particular, on innovation and social objectives
Weak innovation theories do not allow a straighforward interpretation of indicators
A tension: the simplification of policy makers (e.g. the 3% R&D/GDP ratio) versus deepening of analysts
The time dimension: demand of indicators and analyses (quick and dirty) but …. the construction of indicators is a “heavy ship” (more than 10 years to build)
The relationship between producers and users
OECD Group of ‘National Experts on Science and Technology
Indicators’ - NESTI
Producers of indicators
20
NESTI: Working Party of National Experts on Science and Technology Indicators (1962)
Mission:Produce methodologies, statistics and analysesCo-ordinating bodyClearing house
NESTI
21
OECD and member countriesEurostat and the relevant Commission DGs and
member states of the EUUNESCOUnited Nations Institute of StatisticsRICYT and member countriesNEPAD is moving in this direction
International organisations active in STI indicators
22
In the 1960s few countries collected data on S&TThe OECD offered a ready-made model for those who had not yet
developed the necessary instrumentsStandardisation was proposed by an international organisation and
not by a specific countryThe Frascati Manual was introduced with a petits pas strategy
- the first edition was an internal document only (1962)- the Manual was tested (1963-1964) in many countries- it was revised in light of the experience gained from the surveys
The Frascati Manual has never been an imperative document: countries are totally free to apply its conventions
No harmonised questionnaire
The role of OECD in the development of S&T measurement in the 1960s
23
United Nations Fundamental Principles of Official Statistics
Official statistics provide an indispensable element in the information system of a democratic society. They need to be compiled and made available on an impartial basis by official agencies.
To retain trust in official statistics, the statistical agencies need to decide according to strictly professional considerations on methods and procedures.
To facilitate a correct interpretation of the data, the statistical agencies are to present information according to scientific standards on the sources, methods and procedures of the statistics.
Source: UN Statistical commission
24
Data for statistical purposes may be drawn from all types of sources, be they statistical surveys or administrative records.
Individual data collected by statistical agencies are to be strictly confidential and used exclusively for statistical purposes.
The laws, regulations and measures under which the statistical systems operate are to be made public.
Coordination among statistical agencies is essential.
United Nations Fundamental Principles of Official Statistics
Source: UN Statistical commission
25
The “dangerous” business of statisticians
What matters is not how one fashions things, but what one does with them; not the weapon, but the battle (…). The making and the using of the tool are different things.
O. Spenger, Man and Technics: A Contribution to a Philosophy of Life, 1932
26
The members of the indicators community have to play the difficult game of finding a proper balance between data collection, analysis and policy making – preserving their intellectual integrity
Indicators community
27
Models of innovation
Indicators make sense in the context of models
Linear
Chain-linked
Triple helix
Open innovation
Research Development ProductionDesign Engineering
• Based on research
• Sequential
• Technocratic
The Linear model of innovation
•Mercato•potenziale
• Invenzione/•progettazione
• analitica
•Progettazione •dettagliata e
• test
•Distribuzione•e mercato
•Ricerca
Potential
market
Invention/
analytical
project
Detailed
project
& test
Re-design &
productionDistribution
& market
Research
Knowledge
Based on designInteractiveResearch is not a pre-requisite for innovation
The Chain-linked model of innovation
30
The Triple-helix model of innovation
31
The Triple-helix model of innovation
Government
Government-sponsored Research Institutes
Research for public purposes
Applied research
Universities
Supply scientists & engineers
Curiosity-driven
Basic research
Firms
Commercialisation
Produce innovationDevelopment (R&D))
32
Open innovation
33
Open innovation
34
Open innovation
OECD Science, Technology and Industry Scoreboard, 2013
35
Open innovation
OECD Science, Technology and Industry Scoreboard, 2013
36
Open innovation
OECD Science, Technology and Industry Scoreboard, 2013
37
Relationship between research, develoment and innovation
R&D INNOVATION
Applied research
Experimental development
Commercialisation
Distribution
Production
Intellectual property rights
Technical assistance
Basic research
Industralisation
Overlapping area
38
Manuals are codified knowledge
They are guidelines for the collection and interpretation of data and for international comparisons of data, statistics and indicators.
They are supported by an international infrastructure (ISIC, ISCED and ISCO)
Manuals provide a language of discourse and they behave like a technology (our guiding assumption)
What is a Manual?
39
The OECD “Frascati Manual family”
1. The Measurement of Scientific and Technical Activities: Proposed Standard Practice for Surveys of Research and Development. Frascati Manual (2002)
2. Proposed Guidelines for Collecting and Interpreting Technological Innovation Data Oslo Manual (2005)
3. Patent Statistics Manual (1994)4. Manual on the Measurement of Human Resources in
Science and Technology Canberra Manual (1995)5. Proposed Standard Practice for the Collection and
Interpretation of Data on the Technological Balance of Payments (1992)
_______________________________________________
Bibliometrics Manual
40
The OECD “Frascati manual family”
41
Other related manuals and classifications
System of National Accounts – SNA (CEC et al., 1994)
International Standard Industrial Classification – ISIC Rev. 3.1 (UN, 2002)
Statistical Classification of Economic Activities in the European Community – NACE Rev. 1.1 – series 2E.
Nomenclature for the analysis and comparison of scientific programmes and budgets (NABS 2007)
42
The Frascati Manual
43
Freeman: the first edition of the Frascati Manual (1962)
Freeman proposed standardised definitions, concepts, methodologies for conducting R&D surveys and measuring inputs, namely money devoted to R&D and S&T personnel.
Four topics:- definition of research (basic, applied, development)- demarcation with other S&T activities (teaching, production, etc.), - economic sectors (university, government, industry, pnp)- surveying methodology
The underlying model: linear
44
The R&D data
Money
1.Expenditure (GERD)
2.Budget appropriations (GBAORD)
People
45
Research and experimental development (R&D) comprise creative work undertaken on a systematic basis in order to increase the stock of knowledge, including knowledge of man, culture and society, and the use of knowledge to devise new applications
Basic research
Applied research
Experimental development
Frascati Manual: The definition of R&D
46
Frascati Manual: the definition of R&D personnel
R&D personnel All persons employed directly on R&D should be counted, as well as those providing direct services such as R&D managers, administrators, and clerical staff
Researchers
Technicians and equivalent staff
Other supporting staff
47
Oslo Manual
Harmonised questionnaire
48
Types of innovation
Product innovation (goods and services)
Process innovation
Marketing innovation
Organisational innovation
49
Types of innovation
A product innovation is the introduction of a good or service that is new or significantly improved with respect to its characteristics or intended uses. This includes significant improvements in technical specifications, components and materials, incorporated software, user friendliness or other functional characteristics.A process innovation is the implementation of a new or significantly improved production or delivery method. This includes significant changes in techniques, equipment and/or softwareA marketing innovation is the implementation of a new marketing method involving significant changes in product design or packaging, product placement, product promotion or pricing.An organisational innovation is the implementation of a new organisational method in the firm’s business practices, workplace organisation or external relations.
50
Types of innovation
51
A systems approach
Actors: Governments, education and health institutions, business, foreign institutions
Activities: R&D, invention, innovation, diffusion of technologies and practices, HR development
Linkages: Contracts, collaborations, co-publication, grants, monitoring
Outcomes: Wealth, growth, jobsImpacts: Wellbeing, culture change, global influence
The activity of innovation is dynamic, complex, non-linear and global
Innovation: the system approach
52
MonitoringBenchmarkingForesightEvaluationResearch
Use of indicators
53
Monitoring: Comparing the values of a set of indicators over time
Some questions:• How much does the government spend on STI?• Where does it spend it (geography and industry)?• Why does it spend it (socio-economic objectives)?• What does the government get for spending this money?
Some Publications:• European Innovation Scoreboard• OECD STI Scoreboard• OECD Science, Technology and Industry Outlook
Use of indicators
54
Monitoring: Comparing the values of a set of indicators over time
Indicators:• Innovation• R&D• Capital investment• Intellectual property rights• Learning• Education• Design
Use of indicators
55
Benchmarking:
1. Decide upon a set of indicators which are relevant to policy objectives, and then decide about targets to be achieved
2. Agree on a set of indicators and then select another system which is performing better
Examples:Ranking of countries and institutionsThe 3% Lisbon target and ensuing policies (focus the public
debate)
Use of indicators
56
R&D intensity in Italy: the Fata Morgana of 3%
•1970 •1990•1980 •2010 •2020
1.2%
1,5%
3,0%
•1960 •2000 •2030
•Dagli anni 60 ad oggi il rapporto R&S/Pil è rimasto invariato, intorno all’1,1%. Si può prevedere che nei prossimi anni rimarrà agli stessi livelli o che possa aumentare lievemente (l’1,5% teoricamente possibile è rimasto una chimera). L’obiettivo di Lisbona e Barcellona è stato clamorosamente mancato non soltanto in
Italia, ma anche in Europa, anche se è stato riconfermato dai capi di governo per il prossimo futuro.
57
Foresight: viewing the future involving a mix of quantitative and qualitative evaluation methods, including a set of a current set of indicators. Discern likely paths for emerging technologies, possible futures for an economic region, or alternative responses to a coming problem such as the country should function when the oil runs out.
Indicators provide a background to the process but in general are too aggregated
Use of indicators
58
Evaluation: concerns the effective and efficient use allocation of resources in order to achieve a set of objectives
Various methods:Quantitative (bibliometric analysis, turnover from new products,
audits, etc.) and qualitative (peer review)Research Assessment Exercise in the UK (now Research
Excellence Framework VEDERE)
The use of indicators
59
Composite indicators
60
EU Summary Innovation Index
Composite indicator (25 indicators)
3
25
61
The EU Summary Innovation Index
62
The EU innovation performance indicatorSummary Innovation Index
63
Summary Innovation Index (ctd)
64
Summary Innovation Index for EU countries
65
Profile of Italy
66
Global innovation performance
67
Innovation performance of Russia
68
Innovation performance of Russia
In February 2008, the President of the French Republic, Nicholas Sarkozy, asked Joseph Stiglitz , Amartya Sen, and Jean Paul Fitoussi to create a Commission, subsequently called “The Commission on the Measurement of Economic Performance and Social Progress”.
The Commission’s aim has been to identify the limits of GDP as an indicator of economic performance and social progress, including the problems with its measurement; to consider what additional information might be required for the production of more relevant indicators of social progress; to assess the feasibility of alternative measurement tools, and to discuss how to present the statistical information in an appropriate way.
The Stiglitz report
12 domains1.Health2.Education and training3.Work and life balance4.Economic well-being5.Social relationships6.Politics and Institutions7.Security8.Subjective well-being9.Landscape and cultural heritage10.Environment11.Research and innovation12.Quality of services
"Gdp and beyond"
Equitable and sustainable well-being
Features of the composite indicator1.Comparisons across space2.Comparison across time3.Non-substituability on elementary indicators4.Simplicity and transparency in calculation5.Immediate use and easy interpretation of results6.Robustness
Methodology
“Corrected MPI”, Mazziotta M., Pareto A. (2013). A Non-compensatory Composite Index for Measuring Well-being over Time. Cogito. MultidisciplinaryResearch Journal, vol. V, n. 4
"Gdp and beyond"
Equitable and sustainable well-being
1. Intensity of research: Percentage of R&D expenditure on GDP.
2. Propensity to patent: Number of patent applications filed to the European Patent Office (EPO) per million of inhabitants..
3. Impact of knowledge workers on employment: Percentage of persons employed with university education (ISCED 5-6) in scientific-technological occupations (ISCO 2-3) on total persons employed.
4. Innovation rate of the productive system: Percentage of firms that have introduced technological (product or process), organizational or marketing innovation in a three-year period on total number of firms with at least 10 number of persons employed.
Equitable and sustainable well-being
5. Innovation rate of product/service of the national productive system: Percentage of firms that have introduced innovations of product/service in a three-years period on total number of firms with at least 10 number of persons employed .
6. Productive specialization in knowledge-intensive sectors: Percentage of persons employed in high-tech manufacturing sectors and those in knowledge-intensive services on total of persons employed.
7. Intensity of Internet use: Percentage of people aged 16-74 years who have used internet at least once a week during the 12 months before the interview on total people aged 16-74 years.
Equitable and sustainable well-being (ctd)
Ranking of universities
Four major sources
ARWU Shangai (Shangai, Jiao Tong University)QS World University Ranking THE University Ranking (Times Higher Education)US News e World Reports (Best Global Universities)
“Starting from 2003, ARWU has been presenting the world Top 500 universities annually based on a set of objective indicators and third-party data. ARWU has been recognized as the precursor of global university rankings and the most trustworthy league table. ARWU adopts six objective indicators to rank world universities, including:-the number of alumni and staff winning Nobel Prizes and Fields Medals, -the number of Highly Cited Researchers, -the number of articles published in journals of Nature and Science, -the number of articles indexed in Science Citation Index - Expanded and Social Sciences Citation Index, and -per capita performance. More than 1200 universities are actually ranked by ARWU every year and the best 500 universities are published.”
Academic Ranking of World Universities Shanghai Jiao Tong University
Criteria selected as the key pillars of what makes a world class university:ResearchTeachingEmployabilityInternationalisation Facilities Social ResponsibilityInnovationArts & Culture InclusivenessSpecialist Criteria
TopUNIVERSITIES Worldwide university rankings, guides & events
ResearchIndicators considered here include assessments of research quality amongst academics, productivity (i.e. number of papers published), citations (i.e. how recognized and referred to those papers are by other academics) and awards (e.g. Nobel Prizes or Fields Medals).
TeachingA key role of a university is the nurture of tomorrow's finest minds, inspiring the next generation of potential research academics. Typical indicators in teaching quality assessments are collation of student feedback through national student surveys, further study rate and student faculty ratio.
EmployabilityGraduate employability encompasses more than academic strength, focusing on ‘work-readiness’ - the ability to work effectively in a multi-cultural team, to deliver presentations, to manage people and projects. Common indicators in this area are surveys of employers, graduate employment rates and careers service support.
Internationalisation Here, effective indicators could be the proportion of international students and staff, the numbers of exchange students arriving and departing, the number of nationalities represented in the student body, the number and strength of international partnerships with other universities and the presence of religious facilities.
Facilities University infrastructure is an indicator which enables students to know what to expect from their university experience. Indicators such as sporting, IT, library and medical facilities, as well as the number of students societies are considered within this criterion.
Online/Distance learningThis category looks at various indicators such as student services and technology, track record, student faculty engagement, student interaction, commitment to online and reputation of the university.
Social ResponsibilityEngagement measures how seriously a university takes its obligations to society by investing in the local community as well as in charity work and disaster relief. It also analyses the regional human capital development and environmentally awareness.InnovationInnovation, the output of the universities activities and findings to economy, society and culture, has become increasingly relevant for universities.Arts & Culture
Effective indicators are the number of concerts and exhibitions organized by the institution, the number of credits and cultural awards and cultural investment.
InclusivenessThis area looks at the accessibility of the university to students, particularly at scholarships and bursaries, disability access, gender balance and low-income outreach.
Specialist CriteriaExcellence in a narrow field is as valid a claim to world-class status as competence in the round. These criteria are designed to extend credit where it's due. This category looks at accreditations and discipline rankings.
TopUNIVERSITIES Worldwide university rankings, guides & events
Ranking of universities: the case of Italy
ARWU Shangai (Shangai, Jiao Tong University)QS World University Ranking THE University Ranking (Times Higher Education)US News e World Reports (Best Global Universities)
ARWU Shangai: Bologna 173, Milano 186, Padova 188, Pisa 190, Sapienza 191
QS World University Ranking: Bologna 182, Sapienza 202, Politecnico Milano 229
World University Ranking SA: Sapienza 95, Bologna 99, Pisa 184, Milano 193
US News e World Report: Sapienza 139, Bologna 146, Padova 146, Milano 155
OECD Science, Technology and Industry Outlook, 2008
OECD Science, Technology and Industry Outlook, 2008
OECD Science, Technology and Industry Scoreboard, 2013
OECD Science, Technology and Industry Outlook, 2008
OECD Science, Technology and Industry Scoreboard, 2013
OECD Science, Technology and Industry Outlook, 2008
OECD Science, Technology and Industry Scoreboard, 2013
OECD Science, Technology and Industry Outlook, 2008
OECD Science, Technology and Industry Scoreboard, 2013
OECD Science, Technology and Industry Scoreboard, 2013
OECD Science, Technology and Industry Scoreboard, 2013
90
Clusters
91
As OECD admitted: “Monitoring and benchmarking are not coupled with policy evaluation (…). They are seldom used for evaluation purposes (…) but to analyse [counties’] position vis-à-vis competing countries and to motivate adaptation or more intense policy efforts (…).”
“Official statistics mainly served discourse purposes, and in this sense the accounting framework and the statistics presented within it were influential because they fit perfectly well with the policy discourse on rationality, efficiency and accountability: it aligns and frames the science system, by way of statistics, as goal-oriented and accountable. As it actually is, the accounting in official statistics on science is a metaphor, not an accounting exercise as such”
Use of indicators (Benoit Godin)
92
A rhetoric device: a plethora of figures and graphs
“In the various studies on productivity and the New Economy the OECD constantly reminded the reader that the links between science, technology and productivity have not been demonstrated”.
“A large series of graphs and figures could persuade the reader of the seriousness of the study. Although no statistics could be used to prove the emergence of the New Economy, graphs and figures nevertheless served the purpose of empiricism”.
(Godin, 2004)
93
The rhetoric of numbers Policy prescriptions based on shaking statistical evidence
(New Economy)The “umbrella” concept, slogans, buzzwords, which shape
new ways to arrange old indicators
Use of indicators (Benoit Godin)
94
A rhetoric device: a plethora of figures and graphs
“Secure a quantitative statement of the critical elements in an official’s problem, draw it up in concise form, illuminate the tables with a chart or two, bind the memorandum in an attractive cover tied with a neat bow-knot (…). The data must be simple enough to be sent by telegraph and compiled overnight”
(Mitchell, 1919)
95
GERD is used for target setting - from descriptive to prescriptive
“The American GERD/GDP ratio of the early 1960s, that is 3%, as mentioned in the first paragraphs of the first edition of the Frascati Manual, became the ideal to which member countries would aim, and which the OECD would implicitly promote” (Godin)
Lisbona UE 3% (2% business, 1% public sector)
The mystique of ranking
OECD Science, Technology and Industry Outlook, 2008
97
Keith Pavitt
“One would think that the political agenda determines the collection and analysis of indicators. In reality it is the other way round: it is the availability of indicators which steers the political discourse.”
98
Fred Gault
“Policy analysts should be both literate and numerate, able to put a case using innovation indicators. Not only should the analysits have such a skill set, but they also require some knowledge of the subject. It is in this environment that monitoring, benchmarking and evaluation lead to policy learning and to more effective policies.”
99
Advising has become the preserve of applied economists (no more physicists and sociologists)
Advisors need to be – and to appear – experts who guarantee a competent and independent approach
The compromise between engagement and integrity of analysts
Frustration of advisors is part of the gameThe paradox: too many or too little indicators?The pressure of vested interests on official statisticians
The policy advisor
100
Innovation expenditure. The Oslo Manual
R&D = 55%R&D = 38%
101
A dilemma for users
• BERD (Frascati), Annual, all NACE industries, no size threshold
• CIS (Oslo), selected NACE industries, 10 employees or more, sample
102
Research Evaluation
Evaluation of what:researcheducation“third mission” of universities and research agencies (consultancy, support to local authorities, etc.)
Evaluation by whom:experts, peers
Evaluation of what:organisations (departments, universities, schools)programmes, projectsindividuals (professors, researchers, students)
Evaluation whenex-antein-itinereex-post
103
Research Evaluation
Indicators used
- bibliometrics- R&D- peer review- students- graduates- patents- spin-offs- contracts and other funding- other
104
Evaluation in Italy
New public managementAccountabilityValue for money
National Institute for the Evaluation of the Education and Training System
105
San Francisco Declaration on Research Assessment
The Journal Impact Factor, as calculated by Thomson Reuters, was originally created as a tool to help librarians identify journals to purchase, not as a measure of the scientific quality of research in an article.
With that in mind, it is critical to understand that the Journal Impact Factor has a number of well-documented deficiencies as a tool for research assessment. These limitations include:
A) citation distributions within journals are highly skewed; B) the properties of the Journal Impact Factor are field-specific: it
is a composite of multiple, highly diverse article types, including primary research papers and reviews;
C) Journal Impact Factors can be manipulated (or “gamed”) by editorial policy; and
D) data used to calculate the Journal Impact Factors are neither transparent nor openly available to the public.
106
General RecommendationDo not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions.
San Francisco Declaration on Research Assessment
Evaluation should enhance efficiency and effectivenessEvaluation is a difficult processPeer review vs bibliometricsNSE vs SSHUsed to define financing and hiring/promoting staffLeague tablesPro-active evaluation vs punitive evaluationCompetition vs cooperation of scientistsExcellenceOpportunistic behaviour and the split of the academic communityThreat t othe equilibrium amongst the activities of the universityResearch Assessment Excercise (REF) in the UK
Lessons from research evaluation
108
After 50 years STI indicators are quite goodInternational cooperation is key (the role of the OECD and EU)Support to non-OECD countriesFurther develop theoretical models and methodologiesFrom macro to microKeep the dialogue between actors aliveMore expertise: too many economists (philosophers,
sociologists, other social scientists)Too many indicators?Resource constraintsSTI indicators are a public good, and the “owner” of statistics is
the StateThe role of commercial data producers (e.g. bibliometrics)What to do with the misuse of indicators?Producers: stay creative, autonomous, and risk-taking
Concluding remarks
Thank you for your attention