Post on 25-Feb-2016
description
IBEN BRØNDUMAARHUS UNIVERSITY LIBRARYBUSINESS AND SOCIAL SCIENCESIBROENDUM @SAM.AU.DK
18TH NORDIC WORKSHOP ON BIBLIOMETRICS AND RESEARCH
POLICY 28. OCTOBER 2013
RESEARCHERS’ VIEWS ON RESEARCH EVALUATION AND THE DANISH BIBLIOMETRIC RESEARCH INDICATOR
AGENDA›Introduction›Methods›Results
-Influence of main field on attitudes towards research evaluation elements
-Influence of publication activity on attitudes towards DBRI-Themes in respondents’ comments
›Conclusions2
INTRODUCTION›LIS practice and bibliometrics
›DBRI launched in 2009
›Previous studies on the effect of evaluation based funding on publication behaviour (e.g. Butler, 2003; Gläser et al., 2002) and how bibliometrics impact the science system (Weingart, 2005)
›2012: Evaluation of DBRI (Sivertsen & Schneider, 2012)
3
4
OBJECTIVES OF THE SURVEY›To investigate researchers’ attitudes towards research evaluation in the form of h-index, publication and citation counts
›To explore their view on the Danish Bibliometric Research Indicator (DBRI) and how it may have affected their research
METHODS› 400 researchers from 5 major universities
in DK (University of Copenhagen, Aarhus University, University of Southern Denmark, Aalborg University, Roskilde University) 80 from each university
› Systematic random sampling
› Email with link to online questionnaire, short description of the survey and information about anonymity for them and their institution
› 161 respondents = response rate of 40 %
Survey part
Number of respondents
Response rate
Whole survey
161 40 %
Research evaluation
148 37 %
DBRI 159 40 %
Background information
157 39 %
5
REPRESENTATIVENESS OF THE SAMPLEPercentage per main field compared to statistics from Danish Universities (Danske Universiteter, n.d.)
6
Main field Sample Danish Universities
Arts 17 % 17 %Social sciences 22 % 16 %Health 26 % 20 %Science 27 % 46 %Technology 6 %Business 1 % -Not disclosed 1 % -Other - 1 %
ATTITUDES TOWARDS ASPECTS OF RESEARCH EVALUATION - GENERAL
7
H-index as a measure of a researcher's
productivity and im-pact
Publication counts as measures of productiv-
ity
Citation counts as measures of the
impact of publica-tions
0%10%20%30%40%50%60%70%80%90%
100%Positive
Somewhat posi-tive
Neither nor
Somewhat nega-tive
Negative
Do not know
… COMPARED TO MAIN FIELD
8
Positive Somewhat positive Neither nor Somewhat negative Negative0
10
20
30
40
50
60
Publication counts as measures of productivity
Not disclosed
Technology
Health
Social sciences
Science
Arts
Business
… COMPARED TO MAIN FIELD
9
Positive Somewhat positive Neither nor Somewhat negative Negative0
10
20
30
40
50
60
Citation counts as measures of the impact of pub-lications
Not disclosed
Technology
Health
Social sciences
Science
Arts
Business
… COMPARED TO MAIN FIELD
10
Positive Somewhat positive
Neither nor Somewhat negative
Negative Do not know0
5
10
15
20
25
30
35
40
45
50
H-index as a measure of a researcher's productivity and impact
Not disclosed
Technology
Health
Social sciences
Science
Arts
Business
ATTITUDES TOWARDS DBRI
11
… motivates researchers to publish in the
most esteemed and prestigious
publication channels
… strengthens the quality of Danish
research
… increases the exposure of
Danish research
… emphasises the importance of disseminating your research
… has affected my research positively
… has affected my research
negatively
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%The DBRI....
Strongly agree
Somewaht agree
Neither nor
Somewhat disagree
Strongly disagree
Do not know
12
Strongly agree Somewhat agree
Neither nor Somewhat disagree
Strongly disagree
Do not know0
5
10
15
20
25
30
DBRI has affected my research positively
01-2021-4041-6061-8081-100101-200>201Not disclosed
INFLUENCE OF PUBLICATION ACTIVITY – INT. ARTICLES
13
Strongly agree Somewhat agree
Neither nor Somewhat disagree
Strongly disagree
Do not know0
5
10
15
20
25
30
DBRI has affected my research negatively
01-2021-4041-6061-8081-100101-200>201Not disclosed
INFLUENCE OF PUBLICATION ACTIVITY – INT. ARTICLES
COMMENTS ON RESEARCH EVALUATION ASPECTS› One size doesn’t fit all!› Too general and undifferentiated› Disciplines/fields are diverse› Lack of context› Quantity not quality
The three measures are totally dependent on the type of research, the specialty and the size of the subject area,
“sex factor” and the number of persons interested in the subject.
Publication culture and channels differ very much in the various fields. It is impossible to compare quantitatively […]
14
CITATION COUNTS
Numbers of citations can be misleading because of a few high impact papers, a large number of reviews, or co-authorship on high impact papers with little direct involvement by the researcher.
If you work in a smaller research field (e.g. pituitary gland neoplasms) it will be less cited etc. than a broader field
(e.g. diabetes, type 2) – but that does not mean that one type of research is more important than the other.
Researchers with in [the same] network cite each other not only because it is relevant, but because it boosts the citation impact factor. Strategically a sensible action, but does that say anything about quality and impact?
15
H-INDEXThere is not necessarily a connection between a
researcher’s productivity and his/her h-index/number of publications. A researcher’s publications can easily have high impact (i.e. been read by many) and not necessarily been cited for it.
The h-index is not accurate as it does not take the amount of time a person has published into account […]
[…] A researcher, for example, can have a high h-index as co- author on many high impact papers, without having contributed much to the work. A scientist can also publish many [papers] with little or no impact […]
16
‘DISMISSAL’
[…] The numbers are easily manipulated by researchers and the system is grossly exploited by the journals who demand exorbitant prices for publishing scientific articles.
[The three measures] are good for nothing; except for disciplining (the idea is to have as many areas to measure on as possible so that the individual is always behind).
They impact the publication style so that people for example perform minor changes in texts and publish them again […]
17
COMMENTS ON THE DBRI›Adversely affects publication behaviour›DBRI authority file›Societal consequences›‘Dismissal’
18
PUBLICATION BEHAVIOUR
Now I only think of points and no longer of recognition. I intentionally cut my research into bits suitable for DBRI-publication
and I am no longer interested in creating new coherent understanding/knowledge.
DBRI crucially increases the motivation for more slicing […].
DBRI has done nothing but give rise to suboptimisation regarding publication channels. Consequently, people speculate in more publications, not better publications […]
DBRI forces me to publish in prestigious channels, but not in the channels that are read by the people who apply my research […]
19
PUBLICATION BEHAVIOUR
The winning strategy is to publish as much as possible in the lowest of the top 20% journals. […] A second possible winning strategy is to type fast, since the difference between 3 and 1 point is not stark. So you may be able to type three times more 1-point papers and still come out on top.
[…] I do not believe that coercion in relation to publishing and the focus on particular journals increase quality. On the contrary, it is an alignment that pleases the journal publishers’ demands and interests […]
I publish in a more conscious way now, but I do not think (the dissemination of) my research has either improved or worsened because of this – it is just different.
20
DBRI AUTHORITY FILE OF PUBLICATION CHANNELS
[…] A two-level separation as in the Danish system is not sufficient: the very top journals have exponentially more impact and visibility than the lower journals within the top 20% […]
[…] Many journals accepted by DBRI are in my opinion of little value and with extremely low impact […]
[…] The classification of journal levels is enveloped in mystery. It does not always depend on quality, but on where the committee members themselves publish. Therefore, it is crucial for institutions to have persons in these committees so that the journals you yourself publish in are placed in the top-level. 21
The committees make lists permeated by subjective choices and personal interpretations […]
[…] The classification of a journal (level 1 or 2) seems very random – it seems to be decided by where the committee members themselves publish.
[…] A publication is accepted by a level 2 journal one year and when it finally comes out a year later, the very same journal has become a level 1 journal. This motivates going for the low-hanging fruit […] The system [the authority files] should be prospective; everything else is an arbitrary lottery.
22
DBRI AUTHORITY FILE OF PUBLICATION CHANNELS
SOCIETAL CONSEQUENCES
The purpose of the DBRI is to make the researchers manipulate their personal research indicator by producing boring, easy publishable assembly line research instead of obtaining original and innovative results.
[…] In my opinion the state pays for research that high ranking journals can ‘patent’ and after that the state can pay the publishers to provide access to the very same research […]
23
‘DISMISSAL’
DBRI is a very precise measure of absolutely nothing and has been invented to the delight of bookkeepers. The indicator damages Danish research and must be discontinued ASAP.
The system is useless.
I consider DBRI as a necessary evil and loyally participate in the work to ensure ‘damage control’ and fairness.
24
The assessment of research quality requires peer assessment of research quality. This is the way it is done in the world's leading research countries. Why has Denmark opted for this route? […] The only answer must be that they are a job-creation scheme for librarians?
DBRI is in my opinion probably the most stupid initiative in the modern history of Danish research policy. I believe that DBRI will significantly change the publication tradition with increased slicing, increased self- and friend-citations. […] The only positive thing is that the DBRI might increase job security for research librarians.
25
SOME CONCLUSIONS
›No surprises?
›DBRI
›Cautions
›Further research
26
REFERENCESButler, L. (2003). Modifying publication practices in response to funding formulas. Research Evaluation, 12(1), 39-46. doi: 10.3152/147154403781776780
Danske Universiteter. (n.d.). Universiteternes statistiske beredskab: Personale universiteterne 2007-2012. København: Danske Universiteter. Retrieved 01/10. 2013, from http://www.dkuni.dk/Statistik/Universiteternes-statistiske-beredskab
Gläser, J., Laudel, G., Hinze, S., & Butler, L. (2002). Impact of evaluation-based funding on the production of scientific knowledge: What to worry about, and how to find out. Fraunhofer ISI. Retrieved 09/17.2013, from
Sivertsen, G., Schneider, J. (2012). Evaluering av den bibliometriske forskningsindikator. Oslo: Nordisk institutt for studier av innovasjon, forskning og utdanning. Retrieved 09/25.2013, from fivu.dk/forskning-og-innovation/statistik-og-analyser/den-bibliometriske-forskningsindikator/endelig-rapport-august-2012.pdf
Weingart, P. (2005). Impact of bibliometrics upon the science system: Inadvertent consequences? Scientometrics, 62(1), 117-131. doi: 10.1007/s11192-005-0007-7
27