Post on 27-Mar-2015
Valutazione nelle scienze umane e sociali ********************************
Evaluating Social Sciences and Humanities
CNR, Aula Marconi Piazzale Aldo Moro 7, 00185 Roma
4 aprile 2013
Giorgio Sirilli
ISSiRFA - CNR
Giorgio Sirilli
ISSiRFA - CNR
There are many different types of knowledge and knowledge holders. Not all knowledge is scientific, much knowledge is practical and contex specific
Society is interested in its:Production TransmissionUse
Knowledge
Knowledge
New public management
Evaluation
Research Assesment Exercise (RAE-REF)
Bibliometrics
CNR Department SSH
Research evaluation: the ladder
University
teaching
research “third mission”
The learned society
Research agencies
research
problem solvingmanagement
New Public Management
CatchwordsAccountabilityValue for money
“The rationale of performance funding is that funds should flow to institutions where performance is manifest: ‘performing’ institutions should receive more income than lesser performing institutions, which would provide performers with a competitive edge and would stimulate less performing institutions to perform. Output should be rewarded, not input.”
Herbst, 2007
“Performance based research funding systems” The rationale
Performance based research funding systems are national systems of ex-post university research output evaluation used to inform distribution of funding
“Performance based research funding systems”
CriteriaResearch (teaching excluded)Evaluation ex-post (ex-ante excluded)Research outputResearch funding must depend on the results of evaluationNational system (internal university evaluations excluded)
Share of university funding dependent on “Performance-Based Research Funding Systems”
Share of university funding dependent on “Performance-Based Research Funding Systems”
Country Share (%) of what
Australia 6 Total revenue
Italy 2 Block grant
New Zeland 10 Block grant
Norway 2 Total funding
Slovak Republic 15 Total funding
UK 25 Research support
Hicks D., Reseach Policy (2012)
Share of university funding dependent on “Performance-Based Research Funding Systems”
Hicks D., Reseach Policy (2012)
“The distribution of university research funding is something of an illusion”
“It is the competition for prestige that creates powerful incentives within university systems”
“Performance-based research funding systems aim at excellence: they may compromise other important values such as equity and diversity”
Carried out by CNR in 2009-2010Evaluation of structures, not individualsMethodology
- General panel- Panels (26) composed of 156 scientists- Documentation provieded by Institutes- Site visit- Final score
Lessons: “The General Panel recommended that in the future the two groups of scientific areas (natural sciences and engineering, social science and humanities) remain separated. Experience showed in fact that differences in objectives, research methodologies, evaluation criteria suggest to be deal with them separately”
Evaluation of CNR Institutes
VQR (Evaluation of the Quality of Research)Habilitation of university professors
bibliometric areasnon-bibliometric areas
Both exercises largely based on bibliometrics
ANVUR
Knowledge: the bundle
When two types of evaluation do not match
Internationally oriented journal articlesBooksNational journalsEnlightenment literature
Hicks D., Research Policy (2012)
Four types of publications
International journal literature
WoS/Scopus
Books
High impact
National literature
Local context
Enlightenment
Knowledge transfer
Other means of transfer
The four literatures
Hicks D., Research Policy (2012)
Science Social ScienceJournal articles – counted in WoS/Scopus indicators
The 3 other literatures – not counted
Other: books, edited books, book chapters, monographs, reports, creative works
Sources: Bourke et al,1996; Pestana et al., 1995; Winterhager, 1994
81-85%
45-61%
Journal literatureProblem: missing output
Percentage of journal articles indexed in WoS
Ossenblok et al. 2012
Norwegian social science – WoS coverage by field
Ossenblok et al. 2012
Sciences
Genuine scholarly research in any area leads to internationally relevant results
Social sciences are embedded in national contexts, national issues are important, subtle theory is developed in national languages
Academics write in English Social scientists more often publish in their own language
Academics are international in orientation
Social scientists are more oriented to their own national literature than scientists
SSCI coverage of national literatures inadequate
The problem with national literature:Lower level of critique and peer review, questionable quality
National literature
Burnhill & Tubby-Hille, 1994, UK social science
Sciences
Enlightenment literature by field
Research evaluation
NSE SSH
Bibliometrics Helpful Not very helpful
Scientific literature Internationally oriented
Nationally oriented
Peer review Important Very important
Habilitation of professors
Bibliometrics + peer review
Peer review + bibliometrics
Cost Expensive More expensive
Time Limited Longer
Cultural identity15 Institutes and 336 researchers/technologistsSSH
Cultural heritage4 Institutes and 110 researchers/technologistsSSH + NSE
CNR Department of Cultural Identity and Heritage
CNR Departments - Researchers and technologists by training(percentage)
Cultural identity(15 Institutes)
Cultural heritage
(4 Institutes) 33%
38%
46%
42%
10%
3%
18%
10%
Significant differences in specialisation of researchers between former Departments
CNR Departments – Research products (2007-2011)(percentage)
Significant similarities in types of “products” between former Departments
CNR Departments – Research products (2007-2011)(percentage)
Structural difference between SSH Departments and a NSE Department
CNR Departments – Journal articles (2008-2010)
36%
11%
53%
46%
23%
31%
Structural difference between former Departments in terms of output
Contributo in rivistaArticolo in rivistaTraduzione in rivistaNota a sentenzaScheda bibliograficaAbstract in rivistaAbstract/Comunicazione in rivistaAbstract/Poster in rivistaRassegna e recensione Contributo in volumeContributo in volume (capitolo o saggio)Traduzione in volumePrefazione/PostfazioneBreve introduzioneVoce (in dizionario o enciclopedia)Scheda di catalogo Contributo in atti di convegnoContributo in atti di convegnoAbstract/Poster in atti di convegnoComunicazione a convegnoAbstract/Poster in convegnoAbstract/Comunicazione in atti di convegno LibroMonografia o trattato scientificoTraduzione di libroEdizione criticaCommento scientificoConcordanzaIndicePubblicazione di fonti inediteAntologia e manuale scolasticoDizionario e thesaurusBibliografia, repertorio, glossarioCatalogo monograficoManuale tecnico
CuratelaCuratela BrevettoBrevetto Altra tipologiaComposizionePerformanceBanca datiSoftwareCartografiaCommento giuridicoRapporti progetti di ricercaRapporti tecnici, manuali, carte geologiche e tematiche
e prodotti multimedialiRassegne, recensioni, prefazioniAltre pubblicazioniRapporti finali progetti di ricercaMateriale didatticoRapporti di commissioniRapporti statisticiNote interne, manuali e guideProdotti multimedialiRisultati di valorizzazione applicativaProgettiDisegni e designRappresentazioniMostre ed esposizioniManufatti, prototipi d'arte e relativi progettiRapporti tecnici/preprint/working paperData setAltro prodotto
25
Other products: 12 out of 25 (2001-2011)
“Products” for the “market” are marginal: other categories should be added
Market oriented
Academically oriented
International journal literature
Books
National literature Enlightenment
Other means of transfer
CNR products in data base
Evaluation of SSHs: some comments
Knowledge is our concernDo not forget the knowledge bundle and its various facets
2113 0-11.10 Chemist2113 0-11.50 Chemist, analytical2113 0-11.40 Chemist, corrosion2113 0-11.40 Chemist, crystallography2113 0-11.20 Chemist, detergents2113 0-11.20 Chemist, dye2113 0-11.20 Chemist, food2113 0-11.30 Chemist, glass2113 0-11.30 Chemist, inorganic2113 0-11.20 Chemist, leather2113 0-11.30 Chemist, metallurgical2113 0-11.40 Chemist, nuclear2113 0-11.20 Chemist, organic2113 0-11.20 Chemist, paint2113 0-11.20 Chemist, petroleum2113 0-11.90 Chemist, pharmaceutical2113 0-11.40 Chemist, physical2113 0-11.20 Chemist, plastics2113 0-11.20 Chemist, polymer2113 0-11.50 Chemist, quality control2113 0-11.20 Chemist, rubber2113 0-11.20 Chemist, textile
2111 0-12.10 Physicist2111 0-12.50 Physicist, acoustics2111 0-13.50 Physicist, astronomy2111 0-12.80 Physicist, atomic2111 0-12.20 Physicist, ballistics2111 0-12.60 Physicist, electricity and magnetism2111 0-12.70 Physicist, electronics2111 0-12.30 Physicist, heat2111 0-12.20 Physicist, hydrodynamics2111 0-12.40 Physicist, light2111 0-12.20 Physicist, mechanics2111 0-12.80 Physicist, molecular2111 0-12.80 Physicist, nuclear2111 0-12.50 Physicist, optics2111 0-12.20 Physicist, rheology2111 0-12.90 Physicist, solid-state2111 0-12.50 Physicist, sound2111 0-12.90 Physicist, theoretical2111 0-12.30 Physicist, thermodynamics
3415 4-31.20 Representative, sales/technicalXXXXXXX Researcher: MISSING1315 5-10.30 Restaurateur
International Classification of Occupations (ISCO)
22 19
Evaluation of SSHs: some comments
Knowledge is our concernDo not forget the knowledge bundle and its various facetsWhy evaluate?Eveluation is a complex business, and bibliometrics as a
shortcutMuch information on outputs of SSH is not covered by
bibliometricsSSHs are structurally different from NSEs: different types of evaluation
Extending the SSH coverage of literature is OK, but there are inherent limits
Significant differences amongst CNR SSH Department Institutes deserve further analysis
More experimentation needs to be done in SSH: in the meantime be cautiuous
Research evaluation in SSH
Thank you for attention