Measure up! The extent author-level bibliometric indicators are appropriate measures of individual...
-
Upload
lorna-wildgaard -
Category
Leadership & Management
-
view
399 -
download
1
Transcript of Measure up! The extent author-level bibliometric indicators are appropriate measures of individual...
![Page 1: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/1.jpg)
Measure up!The extent author-level bibliometric indicators are
appropriate measures of individual researcher performance
Lorna WildgaardRoyal School of Library and Information Science
Faculty of the Humanities, CPH U27th November 2015
![Page 2: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/2.jpg)
OVERVIEW1. Background
2. Characteristics and effects of ALI (Papers 1, 2, 4)
3. The appropriateness of ALI across different disciplines and different academic seniorities (Papers 3, 5, 6, 7)
4. Concept definintion in the construction of ALI(Chapter 6 & Appendix B, http://tinyurl.com/nj4mvca)
5. Conclusions
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
CORRECTION
![Page 3: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/3.jpg)
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
“ Bibliometrics is the application of variousstatistical analyses to study patterns of authorship, publication, and literature use.” (Lancaster, 1977).
“[…] to analyse the structure of science, measure science, and indicate the production, citations and collaboration of researchers, institutions and countries.” (De Bellis 2014; Pritchard 1969).
![Page 4: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/4.jpg)
4
1980s 1990s
2000s
2010s1970s1960s1950s
de Solla Price growth model
Namilov interdisciplinary
approach to indexing scientific literature
Merton Social structure of scientific communication
Haitun; Yablonsky distribution of citations
Garfield SCI & IF
h-index
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
![Page 5: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/5.jpg)
5
Managerial
Teaching
Outreach
Societal
Technical Skills
Funding/Grants
Productivity
Quality
Relevance
Visibility
Reputation
Scientific
Counting, adding,
dividing, multiplying an authors’ publications & citations
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
![Page 6: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/6.jpg)
DEFINTITIONS OF AU, P & C IN ALIAuthorship, publication and citation is different from discipline to discipline, from time to time and location to location and has implications in indicator development (Bošnjak and Marušić, 2012)
What constitutes a publication [author and citation] should be clearly defined to ensure representative operationalization in the indicator and the extraction of meaningful relationships (Wouters 1999; Skupin 2009; Colledge 2012)
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
![Page 7: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/7.jpg)
EVALUATION
OUTPUT
DATA
ALI MODEL
MOTIVATION TO PUBLISH
MOTIVATION TO CITE
INTERPRETATION OF ALI
LEGITIMATE LINK TO REAL WORLD
PERFORMANCE OF RESEARCHER IN SYSTEM REDEFINES WHAT SUCCESS IS
EXOGENOUS VARIABLES
WHAT IS MISSING
INDEXING POLICY
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
THE PROBLEM
![Page 8: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/8.jpg)
1. What are the characterisitics of ALI of academic performance?
2. To what extent are ALI appropriate in the evaluation of researchers from different disciplines and different academic seniorities?
3. To what extent are the concepts being measured defined in the construction of ALI?
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
RESEARCH QUESTIONS
![Page 9: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/9.jpg)
9
ALI GIVE A SNAP-SHOT OF SELF IMAGE AND CORE PERSONALITY TRAITS
COMPARING RESEARCHERS CAN EXPOSE THE INDIVIDUAL
INDIVIDUALS USE ANY DATA TO INCREASE ALI SCORES, TO INCREASE THEIR SUBJECT VALIDITY & SELF-WORTH
ALI BRING OBJECTIVITY TO THE EVALUATION & REDUCE GENDER; CULTURAL AND RACIAL BIAS
DOCUMENTING BEING OUT-PERFORMED IS DETRIMENTAL TO A RESEARCHER’S SELF-DEFINITION
ALI DO NOT ADD CONTEXT BUT CAN ADD REDUNDANT INFORMATION
SUCCESS IS DEFINED AS WELL IN THE SYSTEM
EXTERNAL CHARACTERISTICS OF ALI
![Page 10: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/10.jpg)
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
INTERNAL CHARACTERISTICS OF ALI
![Page 11: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/11.jpg)
ALI are designed to measure particular aspects of the effect of a researcher’s work – over time, to field, as quality, ranking all or selected works, co-authorship etc.
Judgements based on ALI can lead to assumptions about the productivity and impact of a researcher, which can be unsubstantiated, and affect the psychological character of the individual.
ALI have to be methodologically sound
PRELIMINARY ANALYSIS: RESULTS
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
![Page 12: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/12.jpg)
FAMILY CHARACTERISTICS
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
METHOD: Compute simple indicators on publication and citation data of 750 researchers (585 m, 165w)
4 disciplines, 5 academic senioritiesCorrelation (Kendall’s Tau) as measure of association between seniority and ALI, discipline by discipline ALSCAL IBM SPSS v.19
ENDPOINTSuitability of indicators in different disciplines and seniorities
![Page 13: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/13.jpg)
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
FAMILY CHARACTERISTICS
DISCIPLINE CENTRAL INDICATOR S-STRESS % VARIANCEASTRONOMY hg 0.37 25ENVIRONMENTAL SCIENCE h, h2 0.37 24
PHILOSOPHY IQP 0.38 47PUBLIC HEALTH g 0.49 38
![Page 14: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/14.jpg)
METHOD: Publication and citation data for 512 researchers22,143 papers, 423, 371 citations (WOS)52,227 papers, 746,985 citations (GS)For each researcher 17 ALI calculated
Agreement: determined by matching rank positions Variability of rank position determined by standard deviation of the difference in scholar rank position, calculated from the matched pairs
END POINTS: agreement in ranking between WOS & GS
RANKING CHARACTERISTICS
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
![Page 15: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/15.jpg)
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
hg produced highest level of agreement in rankings in WOS & GS, 0.8-0.9 tau across disciplines and seniorities (Patel 2013)
![Page 16: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/16.jpg)
hg fulfills its potential as a ranking indicator (Alonso et al 2010) yet this tells us nothing about the excellence of the researcher.
h and g cannot be rationally combined to produce indications of research excellence (Franceschini & Maisano 2011).
Indicator rankings inform of the researcher’s visibility in the citation index not in their academic community (Bar-Ilan 2008; De Battisti 2012; Farhadi et al 2013).
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
RANKING CHARACTERISTICS
![Page 17: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/17.jpg)
METHOD: Same data as previous studiesExplorative statistical study of distribution; two step cluster; ordinal regression: odds ratiosIBM SPSS v.22
ENDPOINT: Identify disciplinary & seniority appropriate indicators
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
GROUPING CHARACTERISTICS
![Page 18: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/18.jpg)
CLUSTER 1below IQR, high nnc, sc, low
P & C. Adjusting for academic age mg, m.quotient,
currency of cites improved scores.
CLUSTER 2Median IQR, high scores on
collaboration indicators
CLUSTER 3Top IQR,
highest scores normalized for field, one SIG
paper
CLUSTER 4Extreme outliers,
highest C, cited quickly, highest htype scores
![Page 19: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/19.jpg)
ASTRO HEALTHPHILENVIRO
h2 SUM PP TOP PROP
Q2 e
GROUPING CHARACTERISTICS
Ordinal regression: Academic age is statistically significant for cluster placement Within cluster rank position determined by the ratio P:h
![Page 20: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/20.jpg)
1. Simple ALI are complex!2. Basic descriptive statistics can be more informative3. There are families of indicators4. ALI scores are highly influenced by the database,
academic age and the fit of the data to the model5. ALI are more appropriate in some disciplines than
others 6. Different ALI produce stable rankings within and across
databases.7. ALI can be easily manipulated8. Scores can be due to chance rather than practical
importance.
EMPIRICAL ANALYSIS: CONCLUSIONS
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
![Page 21: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/21.jpg)
Knowing all these practical challenges, what properties should a well constructed indicator contain?
METHOD:Based on papers 1-7Validate ALI using Gingras’ criteriaAnalyze operationalization of AU, P, C in ALI
ENDPOINT:Recommendation of a set of disciplinary dependent indicators of scientific publication output
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
VALIDITY ANALYSIS
![Page 22: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/22.jpg)
THE 3 CRITERIA OF PROFESSOR YVES GINGRAS
1. Adequacy of the indicator to the object it measures -quantity of items a researcher publishes is an indicator of production
2. Sensitivity to the intrinsic inertia of the object -changing values in the time interval correspond to the speed and direction of change in the object
3. Homogeneity of the dimensions of the indicator -h-index is a heterogeneous indicator: it mixes quantity of papers published with citation counts
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
![Page 23: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/23.jpg)
10
28
HOMOGENEITY
ADEQUACY INERTIABackground Preliminary Analyses Empirical Analyses Validation Study Conclusion
17 did not fulfill any of the criteria
n=69 ALI
4 1
720
![Page 24: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/24.jpg)
OPERATIONALIZATION OF AU, P, C”THE LOGIC GRID”
METHOD:• Extract definitions and hypotheses in ALI• Matrix of definitions of AU, P, C & aim of ALI
AIM:Identify in a simple way the properties of ALI
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
![Page 25: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/25.jpg)
HIRSCH QUALITY IMPACT USE OVER TIME IDEA TRANSFER
CITATION
6 6 6 5 2PERFORMANCE EFFECT INFLUENCE POPULARITY REWARD NO DEF.
2 1 1 1 1 20
PUBLISHED SCIENTIST AUTHOR AWARD WINNER
PUBLISHED & CITED SENIORITY NO DEF.
AUTHOR 12 11 10 7 5 2 5
PAPER IN WOS PAPER IN OTHER INDEX PAPER OBJECTS WITH
CITATION EXPRESSION
PUBLICATIONS 30 15 6 1 1
EXCELLENCE EFFECT INDEPENDENCE COMPARE RANK CURRENCY
MEASURE
9 7 7 6 6 4
GROWTH QUALITY DISTRIBUTION QUANTITY & QUALITY CAREER DURABILITY
3 3 2 2 1 1
![Page 26: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/26.jpg)
LOGIC GRID
http://tinyurl.com/ofm7h8s
![Page 27: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/27.jpg)
• ALI are designed for specific author and publication types
• Majority of ALI designed for papers in WoS • Some indicators are purely theoretical• Inherent bias in all ALI is that they are
designed for researcher profiles befitting the hard sciences
• Lack of definitions = methodological deficiency
If developers cannot define the variables they are measuring, how can we?
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
![Page 28: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/28.jpg)
1. TransparencyKnow which data is used to compute the
indicatorUnderstand the math and the inferences
2. DemographicsBe aware if demographics affect the ALI scores
3. MotiveALI must fit the objectives of the evaluation
4. DiversityChoose ALI that fit the discipline
5. OpenessMake the limitations of ALI explicit, use
supplementary ALI
PROPERTIES OF A WELL CONSTRUCTED ALI
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
![Page 29: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/29.jpg)
RQ 1: What are the characterisitics of ALI of academic performance?
ALI are complex in character. They use different:
• disciplinary perspectives, • objectives, • operationalization of variables, • requirements to data, and • mathematical models that favour different disciplines
and seniorities.
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
CONCLUSIONS
one size does NOT fit all.
![Page 30: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/30.jpg)
RQ 2: To what extent are ALI appropriate in the evaluation of researchers from different disciplines and different academic seniorities?
• No seniority ALI were identified
• There is a disconnection between the performance of researcher on the CV and in ALI.
• Some ALI are more appropriate in some disciplines than others as grouping and ranking indicators or as indicators that isolate unique information
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
CONCLUSIONS
![Page 31: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/31.jpg)
RQ 3: To what extent are the concepts being measured defined in the construction of ALI?
• Many indicators are poorly or only partially defined
• Decisions based on these ALI could turn out to be less effective than expected
• If we do not know how the developer has done the measurement, we cannot repeat it, reflect on the meaning associated with the concept and compare results to previous findings
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
CONCLUSIONS
![Page 32: Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance](https://reader035.fdocuments.in/reader035/viewer/2022081605/58ee96f11a28abef7d8b4623/html5/thumbnails/32.jpg)
THANK YOU!!ACUMEN partners: especially Judit Bar-Ilan, Frank van der Most & Paul Wouters.
My supervisors: Jesper Schneider, Peter Ingwersen & Birger Larsen
PhD/Masters group: Jesper, Niels-Peder, Maria, Ole, Rikke, Sara, Sille, Lisa Sutherland & BADASS
Researcher/teacher support: Johanne Maibohm, Susanne Acevedo, Pia Dithmar & Ragnhild Riis.
My boys:Kim, Zander & Balthazar
My opponents: Henk MoedRodrigo Costas & Jeppe Nicolaisen