Rankings workshop How Scopus and SciVal data are used for ... · Guillaume Warnan Senior Customer...
Transcript of Rankings workshop How Scopus and SciVal data are used for ... · Guillaume Warnan Senior Customer...
Guillaume WarnanSenior Customer Consultant, Research Intelligence, Elsevier
Rankings workshop –
How Scopus and SciVal
data are used for global
university rankings
2nd North American SciVal User Group MeetingMcGill University, October 11th 2019
| 1
Overview
➢ WHY are International Rankings important?
➢ WHAT are the International Rankings Organizations?
➢ HOW are SciVal & Scopus involved? (Brief recap overview)
➢ WHICH rankings? Deep dive:
1. QS
2. THE Rankings
➢ EXAMPLE institution data for THE: McGill University
➢ How can data be reproduced from SciVal/ Scopus?
➢ Scopus Institutional Profile Wizard
➢ HOW can citation performance/ranking be improved?
| 2
The importance of Rankings
| 3
Source: University selection by students (IDP Research)
The goal of the program is that by 2020 at
least 5 Russian universities will be in the top
100 universities as denoted by QS World
University ranking.
Rankings are cited by many stakeholders as being
important
85% of students find university ranking as
important in their selection of institute to study
33% of students find university ranking as the
most important factor (number 1 factor, followed by
21% employer recognition, etc.)
Students & Parents
University Management
David Willets (former UK Universities & Science
minister):
“We broadly accept the criteria used by the THE,
which is why our policies are focused on the
same areas.”Policy Makers
| 4
Students primarily use a small number of powerhouse
rankings systems, although there are significant
variations by nationality
Source: Hobsons, INTERNATIONAL STUDENT SURVEY 2015, Value and the Modern International Student
| 5
Academic performance with
league table
Academic performance without
league table
Broad based league table
“Multidimensional” ranking
Employability based league table
Web presence league table
Retroactive
Environmental Focus
Innovation Focus
Global University Rankings
’03 ’04 ’05 ’06 ’07 ’08 ’09 ’10 ’11 ’12 ’13 ’14 ’15 ’16
1 ARWU/Shanghai
2 Webometrics
3 QS
4 4icu
5 NTU/HEEACT
6 Leiden
7 URAP
8 SCImago R R R R R R R
9 THE
10 Trendence/Emerging
11 RUR R R R
12 U-Multirank
13 UI GreenMetric E E E E E
14 CWUR
15 Youth Incorporated
16 nature INDEX
17 RankPro
18 US News
19 Reuters I I
Source: Illuminate Consulting Group
R
E
I
| 6
Other prestigious university rankings where
Scopus is used as bibliometric data source:
National Institutional Ranking Framework
IndiaNational University Ranking
Poland
Best Chinese University Ranking
| 7
All rankings have their strengths and potential
disadvantages and we do not rank the rankings!
• We believe in working on fundamentals with a “basket of
indicators”, always as a complement to peer opinion
• Informed decisions are better decisions
• Metrics should complement, not replace human judgment
• Well-selected metrics drive positive behaviors
• Metrics does not only mean bibliometrics
• Metrics can help monitor and eliminate biases
• Assessments are costly, but availability of new tools help bring
cost down
• Data sources to cover humanities are becoming more
complete
Elsevier’s position on university rankings and metrics
in general
| 8
Brief recap of Scopus and
SciVal
| 9
Scopus is the largest curated abstract and citation database of peer-
reviewed literature, and features smart tools that allow you to track,
analyze and visualize scholarly research.
5000+Publishers
Scopus delivers a comprehensive view on the world of research.
No packages, no add-ons. One all-inclusive subscription.
24,000+
Serial titles
200,000+
Books
76 millionItems
16 millionAuthor profiles
~70,000Affiliation Profiles
1.4 billion cited references
dating back to 1970
Identify and analyze which
journals to read/submit to
Help researchers manage career-
citation counts and h-index
Decide what, where and with
whom to collaborate
Track impact of research;
monitor global research trends
Find out what already exists in
the global world of research
Determine how to differentiate
research topics, find ideas
| 10
As a primary publisher and information aggregator, Elsevier understands the needs of Authors, Editors and Publishers and
provides resources to support the community. Available resources to help journals with successful title review process:
publication ethics resources | FAQs | advisory documents | reviewer comments | editor and publishing services
https://www.elsevier.com/solutions/scopus/content/content-policy-and-selection or [email protected]
Coverage of high quality journals via selection by the
independent Content Selection & Advisory Board (CSAB)
The CSAB is an independent board of subject experts from all over the world.
Board members are chosen for their expertise in specific subject areas; many have (journal) Editor experience.
| 11
Unbiased, comprehensive journal coverage with titles
from many reputable scholarly publishers
Source: May 2016 title list at https://www.elsevier.com/solutions/scopus/content
Other
60%
7%
5%
4%
2%
1%
1%
1%
1%
1%
1%
1%
1%
11%
1%
| 12
Global Representation means global discovery
Across all subjects and content types
North
America
6,000+50% more
than nearest
competitor
Middle East
& Africa
750+212% more
than nearest
competitor
Western
Europe
11,000+69% more
than nearest
competitor
East Europe
incl. Russia
1,400+168% more
than nearest
competitor
Latin
America
700+168% more
than nearest
competitor
Asia
Pacific
2,000+230% more
than nearest
competitor
Australia/
New Zealand
300+206% more
than nearest
competitor
Global
Representation (number of titles)
| 13
And Analyzed in SciVal
SciVal offers quick, easy access to the research performance of
15,500 research institutions and 231 nations worldwide.
| 14
QS
| 15
NEW
Source: QS
Affiliations: Primarily Scopus institutions
Support: Self-citations, subject mapping, etc.
| 16
Source: QS
| 17
Bibliometric parameters
Institutions based
compiled by QS
based on Scopus profiles
NTCC (= Normalized Total
Citation Count) as the core
citation metric
Subject mapping to
ASJC developed
by QS
Scopus data April
2019 snapshot
| 18
Main changes between 2015 and 2018 in QS’ WURM
etr
ics
calc
ula
tion
End Q1 Metr
ics
calc
ula
tion
Q3 (~June/July)
Metr
ics
calc
ula
tion
Q2 (April) Metr
ics
calc
ula
tion
Unchanged
Z-s
core Calculated for the
entire population
Z-s
core Using a fixed range
for z-score calculations to limit divergent effects
Z-s
core
Unchanged
Z-s
core Unchanged
Kilo
papers > 10 affiliations
excluded
Kilo
papers Variable threshold
per subject implemented
Kilo
papers
Unchanged Kilo
papers
Unchanged
Cita
tio
n
Imp
act
NTCC
(introduced in 2015 for the first time)
Cita
tio
n
Imp
act
Unchanged
Cita
tio
n
Imp
act
Unchanged
Citation
Impact
Unchanged
Docum
ent
types
AR RE CP BK CH AIP BZ
Docum
ent
types
UnchangedD
ocum
ent
types
Unchanged
Docum
ent
types
Unchanged
Num
be
r o
f u
niv
ers
itie
s
890 Num
be
r o
f U
niv
ers
itie
s
916 Num
be
r o
f U
niv
ers
itie
s
979 Num
be
r o
f U
niv
ers
itie
s
1,021
Ma
in
Su
bje
cts
5
Ma
in
Su
bje
cts Unchanged
Ma
in
Su
bje
cts Unchanged M
ain
S
ubje
cts
Unchanged
2015 20172016 2018
| 19
THE
| 20
Affiliations: Primarily SciVal institutions
Support:
• Customized bibliometric data from Scopus: Citations (30% of score),
research productivity (6%) & international collaboration (2.5%)
• Reputation data: Elsevier runs the reputation survey (15% of score)
using Elsevier author list for THE
• Affiliation handling: Affiliation corrections, mergers, splits, etc. handled
with THE for the respective universities
Source: THE
| 21
Source: THEElsevier contributes by providing Scopus
bibliometric data
Elsevier contributes by conducting the
reputation survey: Randomly selected authors
from Scopus are invited to complete the survey
| 22
Minimum requirements in order to be ranked
Institutional Data
uploaded through the THE data
portal
1,000 papers for the 5 years
window
At least 150 papers per
year
Variable* threshold
per subject area
(*): 500 papers for large disciplines, 250 for smaller disciplines
TITLE OF PRESENTATION| 23
23|
Main changes between 2015 and 2018 in THE’s WURT
hre
sh
old
o
f p
ap
ers
p
er
ye
ar
200
Th
resh
old
o
f p
ap
ers
p
er
ye
ar
150
Th
resh
old
o
f p
ap
ers
p
er
ye
ar
Unchanged
Th
resh
old
o
f p
ap
ers
p
er
ye
ar
Unchanged
Th
resh
old
p
er
su
bje
ct
0
Th
resh
old
p
er
su
bje
ct
500/250
Th
resh
old
p
er
su
bje
ct
Unchanged
Th
resh
old
p
er
su
bje
ct
Unchanged
Kilo
papers
Excluded Kilo
papers Re-introduced but
with fractionalized counting
Kilo
papers
Unchanged Kilo
papers
Unchanged
Jo
urn
als All
Jo
urn
als Suspended
titles excluded
Jo
urn
als
Unchanged
Jo
urn
als
Unchanged
Docum
ent
types
AR, RE and CP D
ocum
ent
types
AR, RE, CP, BK and CH D
ocum
ent
types
Unchanged
Docum
ent
types
Unchanged
Num
be
r o
f u
niv
ers
itie
s
801 Num
be
r o
f u
niv
ers
itie
s
980 Num
be
r o
f u
niv
ers
itie
s
1,103 Num
be
r o
f u
niv
ers
itie
s
1,258
Su
bje
cts 6
Su
bje
ct
8
Su
bje
ct
11 main disciplines + 32 sub-disciplines
Su
bje
ct
Unchanged
2015 20172016 2018
TITLE OF PRESENTATION| 24
24|
Full description of THE’s methodology to calculate the World University Rankings:
https://www.timeshighereducation.com/world-university-rankings/world-
university-rankings-2020-methodology
TITLE OF PRESENTATION| 25
25|
How to easily find THE’s methodology page:
TITLE OF PRESENTATION| 26
26|
Last year, THE launched a new ranking, the University Impact Rankings
TITLE OF PRESENTATION| 27
27|
Methodology highlights
Only 11 SDGs
evaluated out the 17
To be in the overall
ranking, universities
need to submit data on at least 3
SDGs + SDG 17
Each SDG has a
specific series of metrics
Final score is calculated
by combining its score in
SDG 17 with its top
three scores
TITLE OF PRESENTATION| 28
28|
Elsevier data is being used to assess the “Research” criteria
• Research metrics are derived from data supplied by Elsevier.
• For each SDG, a specific query has been created that narrows the scope of the metric to papers relevant to that SDG.
• As with the World University Rankings, THE is using a five-year window (for the 2019 ranking, 2013-2017). The only exception is the metric on patents that cite research under SDG 9, which relates to the timeframe in which the patents were published rather than the timeframe of the research itself.
• The metrics chosen for the bibliometrics differ by SDG and there are always at least two bibliometric measures used.
TITLE OF PRESENTATION| 29
29|
The weighting of the Research criteria as well as the metrics used vary from one SDG to another
Weighting
# metrics
used
27% 27% 27% 27% 11.6%* 27% 27% 27% 27% 27% 27%
3 3 3 2 1 3 3 3 3 3 2
Main
metric
2 (10%)
3 (10%)
6 (7%)
* + 15.4% on patents, i.e. the number of patents that cite research conducted by the university
Metrics used:
1. Proportion of research papers in the top 10% of journals as defined by Citescore
2. Proportion of research papers that are viewed or downloaded
3. Proportion of research papers that are cited in clinical guidance
4. Proportion of a university’s total research output that is authored by women
1 (10%)
2 (10%)
6 (7%)
1 (10%)
4 (10%)
6 (7%)
1 (14%)
6 (13%)
6 (12%) 1 (10%)
5 (10%)
6 (7%)
1 (10%)
5 (10%)
6 (7%)
1 (10%)
5 (10%)
6 (7%)
1 (10%)
5 (10%)
6 (7%)
1 (10%)
5 (10%)
6 (7%)
7 (14%)
8 (13%)
5. Field-weighted citation index of papers produced by the university
6. Number of publications related to the SDG
7. Proportion of academic publications with co-author from other country
8. Number of publications that relate to the 11 SDGs
| 30
Example of how Scopus data is
prepared for THE’s World
University Ranking
| 31
Process
Reputation
Oct-Nov Dec-Mar Ongoing-May
Preparation for the
reputation survey:
- Questionnaire
- Random selection
of survey invitees
(Scopus author
profiles)
- List of institutions
for selection (all
SciVal institutions)
Reputation Survey
run:
- Regular update to
THE re. response
rate
- Addressing
possible issues
(e.g. emails block)
Bibliometrics
Affiliation validation
and clean up
- THE list against
Elsevier (re)mapped
and checked for
completeness
- SciVal institutions are
default Elsevier profiles
- If not in SciVal, then
added into SciVal or
otherwise from Scopus
- Ongoing feedback
processing from
institutions
- Potential methodology
changes tested and
validated
May-July
Bibliometric
computation:
- Using May-finalized
list, the
bibliometrics:
Citation Score, total
# of papers and
total # of
international papers
- Analysis of the
results, validation
and sign-off
WURLaunch
Sep
World
University
Ranking
launch
27/07/2012
Milestone DescriptionReputation Data
(responses) delivery
27/07/2012
Milestone DescriptionBibliometric Data Delivery
| 32
THE’s WUR 2017-18 Scopus Citation dataset
ALL SCOPUS 2012-2016 Articles,
Reviews, Conf.
papers, Books
and Book
chapters
Disciontinued
titles carved out
12,431,51412,737,67814,061,95667,775,649
Scopus – 1 May 2017
| 34
Example of a citation score calculation
the_name country
rank_
label
overall_
score
teaching
_score
research
_score
citation
_score
industry
_score
internation
al_score
Paris Sciences et Lettres
Ð PSL Research
University Paris France 72 65.2 57 52 85 48.2 76.7
PSL Research University (Paris
Sciences Et Lettres Research
University) publications fwci pubs_intl fwci_intl normdist
Citation
Score
Country Normalized 34185.132 1.476 18987.132 2.137 86.6785.0
Non-normalized 34185.132 1.705 18987.132 2.137 83.32
| 36
Example of THE citation data
from Scopus/SciVal for a
Canadian university
| 37
How THE citation data are calculated for McGill
University...
| 38
Publications of McGill University in SciVal, 2014-2018....
however, not all pub types are eligible...
| 39
Now let’s look at which Scopus profiles this SciVal
profile is sourced from!
| 40
The data for McGill University Scival profile are drawn
from 21 Scopus Institution profiles:
| 41
Scopus profile of McGill University
| 42
Detailed hierarchy of McGill University Scopus profile
TITLE OF PRESENTATION| 43
43|
to analyze more closely the data used by THE?
1
2
3
Use the affiliations from the SciVal profile
Limit to the
document types
eligible for THE
Limit to the past 5 years
(2014-2018)
TITLE OF PRESENTATION| 44
44|
Finally, also add the discontinued titles to the Scopus Advanced Search with: AND NOT(...)
TITLE OF PRESENTATION| 45
45|
You can then re-import your data in SciVal
TITLE OF PRESENTATION| 46
46|
https://www.elsevier.com/solutions/scopus/how-scopus-works/content
Where to find the most recent Scopus discontinued titles list:
TITLE OF PRESENTATION| 47
47|
Use source IDs for your query
TITLE OF PRESENTATION| 48
48|
Excellent new white paper (June 2019) by CSAB member Karen Holland et al. – on the rationale behind the Scopus discontinued titles process and details of how titles are identified:
https://www.elsevier.com/research-intelligence/resource-library/scopus-high-quality-content
...and see recent webinar, Sep. 11 2019 : Scopus Curation and Re-Evaluation = High Quality Content
https://blog.scopus.com/webinars
| 49
Further “VIP” support re. how Scopus/ SciVal data are
used for the World University Rankings – outside the
“regular” Scopus helpdesk channels:
(Disclaimer: as this email address is managed by only 1-2
colleagues within the Scopus product team, and resources
are therefore limited, please only send emails to this
address for any questions for which you cannot find the
information yourself, either on our Scopus website
https://www.elsevier.com/solutions/scopus/how-scopus-
works/content, on THE’s website
https://www.timeshighereducation.com/world-university-
rankings/world-university-rankings-2020-methodology or
within this slide deck.)
| 50Open Access |
User Guide for the Scopus
Institution Profile Wizard (IPW)
| 51Open Access |
How does an institute access the IPW?
• The Scopus IPW was launched in July 2018
• Use is restricted to named authorized users selected by the institution. Normally this should
not be more than three users per institution
• Selected IPW users from an organization should have:
o Familiarity with Scopus.com
o Knowledge of the organization’s hierarchical structure
o Knowledge of all campus and satellite locations and the names they are known by
• A form for the institution to provide authorization of their users has been prepared by the
Elsevier legal team and can be obtained from Elsevier customer consultants
• The signed letter of authorization for the users should be sent along with their individual Scopus
usernames to your Customer Consultant
Important: IPW users will need to register their Scopus account if they have not yet done
so, and log in on Scopus.com at least once before access can be granted
51
| 52Open Access |
Where can I find the IPW on Scopus?
• If you are an authorized IPW user, log in to Scopus
• From the Affiliation search, find your organization
• Open the Affiliation details page. The Modify institution profile link opens the wizard
| 53Open Access |
Function 1: Modify hierarchy
• Click start on the ‘Modify hierarchy’ tile to review and edit the relationships within your
institution
53
(For more detailed instructions, also on the other functionalities within the
Scopus Institutional Profile Wizard, please see the slides in the Appendix.)
| 54
What can an institution do to
improve its Citation Score and
THE ranking?
| 55
What does FWCI represent?
FWCI = Field-Weighted Citation Impact indicates how the number of
citations received by an entity’s publications compares with the
average number of citations received by all other similar publications in
the data universe
A FWCI of 1.00 indicates that the entity’s publications have been cited
exactly as would be expected based on the global average for similar
publications
THE’s Citation Score is based on the normal distribution (Z-score)
of the FWCI population of ALL ranked institutions
| 56
Canadian universities: FWCI & International
Collaboration (2012-2018)
Source: SciVal (data cut of 08/30/19); sample of 48 Canadian universities
| 57
How does international collaboration affect
institutional FWCI on a global scale?
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
FW
CI
% INTERNATIONAL
Asia World
* FWCI for the top 100 Asian institutions by output from each region vs the top 500 international institutions by output
Source: Scopus.com, May 1, 2018
| 58
Impact of Corporate Collaboration for
Universities
Ben Shneiderman, Twin-Win Model: A human-centered approach to research success, Proceedings of
the National Academy of Sciences (2018). DOI: 10.1073/pnas.1802918115
| 59
www.elsevier.com/research-intelligence
Any question?
| 60
Appendix (FWCI methodology)
| 61
Recap on the Citations Score indicator methodology
• The Citation Score for a university is the average of the Z-scores of the
Non-normalized FWCI and the Country normalized FWCI of the university.
More on Z-score here at: https://nl.wikipedia.org/wiki/Z-score
• This means the following:
𝐶𝑖𝑡𝑎𝑡𝑖𝑜𝑛 𝑆𝑐𝑜𝑟𝑒 =𝑁𝑜𝑛−𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 𝐶𝑖𝑡𝑎𝑡𝑖𝑜𝑛 𝑆𝑐𝑜𝑟𝑒 + 𝐶𝑜𝑢𝑛𝑡𝑟𝑦−𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 𝐶𝑖𝑡𝑎𝑡𝑖𝑜𝑛 𝑆𝑐𝑜𝑟𝑒
2
𝑁𝑜𝑛 − 𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 𝐶𝑖𝑡𝑎𝑡𝑖𝑜𝑛 𝑆𝑐𝑜𝑟𝑒 = 𝑍𝑠𝑐𝑜𝑟𝑒 (𝐹𝑊𝐶𝐼)
𝐶𝑜𝑢𝑛𝑡𝑟𝑦 − 𝑁𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 𝐶𝑖𝑡𝑎𝑡𝑖𝑜𝑛 𝑆𝑐𝑜𝑟𝑒 = 𝑍𝑠𝑐𝑜𝑟𝑒 (𝐹𝑊𝐶𝐼
𝐹𝑊𝐶𝐼_𝑐𝑜𝑢𝑛𝑡𝑟𝑦)
• For the FWCI calculation methodology please refer to Appendix F7
| 62
FWCI: the formula
The Field-Weighted Citation Impact (FWCI) for a set of N publications is
defined as:
FWCI ≡1
𝑁
𝑖=1
𝑁𝑐𝑖𝑒𝑖
• 𝑐𝑖 = citations received by publication i
• 𝑒𝑖 = expected number of citations received by all similar publications in the publication
year plus following 3 years
When a similar publication is allocated to more than 1 discipline, the
harmonic mean is used to calculate 𝑒𝑖. For a publication k that is part of 2
disciplines:1
𝑒𝑘≡1
2(1
𝑒𝐴+
1
𝑒𝐵)
𝑒𝐴, 𝑒𝐵 = fractional counts of publications and citations, so that publication k will be counted as
0.5 publications in each of 𝑒𝐴 and 𝑒𝐵, and the citations it has received will also be shared
between A and B.
Field-Weighted Citation Impact is an indicator of mean citation impact, and compares the actual number of citations received by an article with the expected number of citations for articles of the same document type (article, review or conference proceeding paper), publication year and subject field*.
| 63
Example
Publication Discipline Citations
received
Expected citations for
discipline
FWCI
1 Biophysics 5 (5+6+(50/2))/(1+1+(1/2)) = 14.4 (5/14.4) = 0.35
2 Biophysics 6 (5+6+(50/2))/(1+1+(1/2)) = 14.4 (6/14.4) = 0.42
3 Chemistry 120 (120+130+(50/2)/(1+1+(1/2)) =
110.0
(120/110.0) = 1.09
4 Chemistry 130 (120+130+(50/2)/(1+1+(1/2)) =
110.0
(130/110.0) = 1.18
5 Biophysics
and
Chemistry
50 Harmonic mean of citations
received, when publication is in
more than 1 discipline:
2/((1/110) + (1/14.4)) = 25.46
(50/25.46) = 1.96
FWCI database of 5
publications:
((0.35+0.42+1.09+1.18+1.96)/5
) = 1.00
FWCI Biophysics ((0.35+0.42+1.96)/3) = 0.91
FWCI Chemistry ((1.09+1.18+1.96)/3) = 1.41
| 64Open Access |
T&F book published in 2017: overview of all worldwide
rankings & main bibliographic databases used
https://www.taylorfrancis.com/books/9781315155890
(In case you would like to receive a PDF copy of a preprint of our Scopus book
chapter, please send me an email at [email protected])