Valutazione nelle scienze umane e sociali ********************************

Evaluating Social Sciences and Humanities

CNR, Aula Marconi Piazzale Aldo Moro 7, 00185 Roma

4 aprile 2013

Giorgio Sirilli

ISSiRFA - CNR

Giorgio Sirilli

ISSiRFA - CNR

Our subject: Evaluation

Knowledge: the bundle

There are many different types of knowledge and knowledge holders. Not all knowledge is scientific, much knowledge is practical and contex specific

Society is interested in its:Production TransmissionUse

Knowledge

Knowledge

New public management

Evaluation

Research Assesment Exercise (RAE-REF)

Bibliometrics

CNR Department SSH

Research evaluation: the ladder

University

teaching

research “third mission”

The learned society

Research agencies

research

problem solvingmanagement

New Public Management

CatchwordsAccountabilityValue for money

“The rationale of performance funding is that funds should flow to institutions where performance is manifest: ‘performing’ institutions should receive more income than lesser performing institutions, which would provide performers with a competitive edge and would stimulate less performing institutions to perform. Output should be rewarded, not input.”

Herbst, 2007

“Performance based research funding systems” The rationale

Performance based research funding systems are national systems of ex-post university research output evaluation used to inform distribution of funding

“Performance based research funding systems”

CriteriaResearch (teaching excluded)Evaluation ex-post (ex-ante excluded)Research outputResearch funding must depend on the results of evaluationNational system (internal university evaluations excluded)

Share of university funding dependent on “Performance-Based Research Funding Systems”

Share of university funding dependent on “Performance-Based Research Funding Systems”

Country Share (%) of what

Australia 6 Total revenue

Italy 2 Block grant

New Zeland 10 Block grant

Norway 2 Total funding

Slovak Republic 15 Total funding

UK 25 Research support

Hicks D., Reseach Policy (2012)

Share of university funding dependent on “Performance-Based Research Funding Systems”

Hicks D., Reseach Policy (2012)

“The distribution of university research funding is something of an illusion”

“It is the competition for prestige that creates powerful incentives within university systems”

“Performance-based research funding systems aim at excellence: they may compromise other important values such as equity and diversity”

Carried out by CNR in 2009-2010Evaluation of structures, not individualsMethodology

- General panel- Panels (26) composed of 156 scientists- Documentation provieded by Institutes- Site visit- Final score

Lessons: “The General Panel recommended that in the future the two groups of scientific areas (natural sciences and engineering, social science and humanities) remain separated. Experience showed in fact that differences in objectives, research methodologies, evaluation criteria suggest to be deal with them separately”

Evaluation of CNR Institutes

VQR (Evaluation of the Quality of Research)Habilitation of university professors

bibliometric areasnon-bibliometric areas

Both exercises largely based on bibliometrics

ANVUR

Knowledge: the bundle

When two types of evaluation do not match

Internationally oriented journal articlesBooksNational journalsEnlightenment literature

Hicks D., Research Policy (2012)

Four types of publications

International journal literature

WoS/Scopus

Books

High impact

National literature

Local context

Enlightenment

Knowledge transfer

Other means of transfer

The four literatures

Hicks D., Research Policy (2012)

Science Social ScienceJournal articles – counted in WoS/Scopus indicators

The 3 other literatures – not counted

Other: books, edited books, book chapters, monographs, reports, creative works

Sources: Bourke et al,1996; Pestana et al., 1995; Winterhager, 1994

81-85%

45-61%

Journal literatureProblem: missing output

Percentage of journal articles indexed in WoS

Ossenblok et al. 2012

Norwegian social science – WoS coverage by field

Ossenblok et al. 2012

Sciences

Genuine scholarly research in any area leads to internationally relevant results

Social sciences are embedded in national contexts, national issues are important, subtle theory is developed in national languages

Academics write in English Social scientists more often publish in their own language

Academics are international in orientation

Social scientists are more oriented to their own national literature than scientists

SSCI coverage of national literatures inadequate

The problem with national literature:Lower level of critique and peer review, questionable quality

National literature

Burnhill & Tubby-Hille, 1994, UK social science

Sciences

Enlightenment literature by field

Research evaluation

NSE SSH

Bibliometrics Helpful Not very helpful

Scientific literature Internationally oriented

Nationally oriented

Peer review Important Very important

Habilitation of professors

Bibliometrics + peer review

Peer review + bibliometrics

Cost Expensive More expensive

Time Limited Longer

Cultural identity15 Institutes and 336 researchers/technologistsSSH

Cultural heritage4 Institutes and 110 researchers/technologistsSSH + NSE

CNR Department of Cultural Identity and Heritage

CNR Departments - Researchers and technologists by training(percentage)

Cultural identity(15 Institutes)

Cultural heritage

(4 Institutes) 33%

38%

46%

42%

10%

3%

18%

10%

Significant differences in specialisation of researchers between former Departments

CNR Departments – Research products (2007-2011)(percentage)

Significant similarities in types of “products” between former Departments

CNR Departments – Research products (2007-2011)(percentage)

Structural difference between SSH Departments and a NSE Department

CNR Departments – Journal articles (2008-2010)

36%

11%

53%

46%

23%

31%

Structural difference between former Departments in terms of output

Contributo in rivistaArticolo in rivistaTraduzione in rivistaNota a sentenzaScheda bibliograficaAbstract in rivistaAbstract/Comunicazione in rivistaAbstract/Poster in rivistaRassegna e recensione Contributo in volumeContributo in volume (capitolo o saggio)Traduzione in volumePrefazione/PostfazioneBreve introduzioneVoce (in dizionario o enciclopedia)Scheda di catalogo Contributo in atti di convegnoContributo in atti di convegnoAbstract/Poster in atti di convegnoComunicazione a convegnoAbstract/Poster in convegnoAbstract/Comunicazione in atti di convegno LibroMonografia o trattato scientificoTraduzione di libroEdizione criticaCommento scientificoConcordanzaIndicePubblicazione di fonti inediteAntologia e manuale scolasticoDizionario e thesaurusBibliografia, repertorio, glossarioCatalogo monograficoManuale tecnico  

CuratelaCuratela BrevettoBrevetto Altra tipologiaComposizionePerformanceBanca datiSoftwareCartografiaCommento giuridicoRapporti progetti di ricercaRapporti tecnici, manuali, carte geologiche e tematiche

e prodotti multimedialiRassegne, recensioni, prefazioniAltre pubblicazioniRapporti finali progetti di ricercaMateriale didatticoRapporti di commissioniRapporti statisticiNote interne, manuali e guideProdotti multimedialiRisultati di valorizzazione applicativaProgettiDisegni e designRappresentazioniMostre ed esposizioniManufatti, prototipi d'arte e relativi progettiRapporti tecnici/preprint/working paperData setAltro prodotto

25

Other products: 12 out of 25 (2001-2011)

“Products” for the “market” are marginal: other categories should be added

Market oriented

Academically oriented

International journal literature

Books

National literature Enlightenment

Other means of transfer

CNR products in data base

Evaluation of SSHs: some comments

Knowledge is our concernDo not forget the knowledge bundle and its various facets

2113 0-11.10 Chemist2113 0-11.50 Chemist, analytical2113 0-11.40 Chemist, corrosion2113 0-11.40 Chemist, crystallography2113 0-11.20 Chemist, detergents2113 0-11.20 Chemist, dye2113 0-11.20 Chemist, food2113 0-11.30 Chemist, glass2113 0-11.30 Chemist, inorganic2113 0-11.20 Chemist, leather2113 0-11.30 Chemist, metallurgical2113 0-11.40 Chemist, nuclear2113 0-11.20 Chemist, organic2113 0-11.20 Chemist, paint2113 0-11.20 Chemist, petroleum2113 0-11.90 Chemist, pharmaceutical2113 0-11.40 Chemist, physical2113 0-11.20 Chemist, plastics2113 0-11.20 Chemist, polymer2113 0-11.50 Chemist, quality control2113 0-11.20 Chemist, rubber2113 0-11.20 Chemist, textile

2111 0-12.10 Physicist2111 0-12.50 Physicist, acoustics2111 0-13.50 Physicist, astronomy2111 0-12.80 Physicist, atomic2111 0-12.20 Physicist, ballistics2111 0-12.60 Physicist, electricity and magnetism2111 0-12.70 Physicist, electronics2111 0-12.30 Physicist, heat2111 0-12.20 Physicist, hydrodynamics2111 0-12.40 Physicist, light2111 0-12.20 Physicist, mechanics2111 0-12.80 Physicist, molecular2111 0-12.80 Physicist, nuclear2111 0-12.50 Physicist, optics2111 0-12.20 Physicist, rheology2111 0-12.90 Physicist, solid-state2111 0-12.50 Physicist, sound2111 0-12.90 Physicist, theoretical2111 0-12.30 Physicist, thermodynamics

3415 4-31.20 Representative, sales/technicalXXXXXXX Researcher: MISSING1315 5-10.30 Restaurateur

International Classification of Occupations (ISCO)

22 19

Evaluation of SSHs: some comments

Knowledge is our concernDo not forget the knowledge bundle and its various facetsWhy evaluate?Eveluation is a complex business, and bibliometrics as a

shortcutMuch information on outputs of SSH is not covered by

bibliometricsSSHs are structurally different from NSEs: different types of evaluation

Extending the SSH coverage of literature is OK, but there are inherent limits

Significant differences amongst CNR SSH Department Institutes deserve further analysis

More experimentation needs to be done in SSH: in the meantime be cautiuous

Research evaluation in SSH

Thank you for attention