Scopus’ Re-evaluation Program
Susanne SteigingaSr. Product Manager Scopus ContentASEP, Moscow, April 2019
Content curation and research integrity at its best
Agenda
• Predatory journals – what are they?
• Ongoing content curation via Scopus’ Re-evaluation program
• The pink elephant in the room
• Q&A
PREDATORY JOURNALS
Low quality or ‘predatory journals’
• ‘Predatory journals’ are ill defined and it can be difficult to determine whether a journal is predatory or not.
• There is controversy as to which journals are considered predatory. In many cases there is a lack of factual evidence and no guidance for that view.
• It is not always straightforward to identify predatory journals, because journals change their editorial policies over time and a journal that did not start off as predatory may become so over the years and vice versa.
• Low quality journals can be easier to identify, but some which may, from a bibliometric perspective, seem of low impact play a role in niche areas or serve a local community.
Therefore it is key that all journals in Scopus arethoroughly analyzed manually by the CSAB. This to
ensure the right decisions are made.
How does this work?
First of all: content enters Scopus after rigorous evaluation by the CSAB based on strict selection criteria
Content Selection and Advisory Board (CSAB)
Once a title is indexed in Scopus, the journal performance will be monitored on an ongoing basis
Structural approach: Ongoing content curation to ensure continuous high quality content
Identification of under performing journals
using metrics and benchmarks
Machine learning tool to identify journals with outlier performance
Direct feedback from users and stakeholders
on publication concerns
Re-evaluation by the Content Selection & Advisory Board (CSAB)
Content Curation
Ongoing curation of the full journal base is expected and essential to keep journals at high quality levels and keep the database trustable
Review:
Curate:
Set metrics and benchmarks all journals have to meet every year. If they do not meet any of these benchmarks for two consecutive years, the journal will be re-evaluated by the CSAB
For more info: https://blog.scopus.com/posts/scopus-checks-balances-maintaining-quality-content-on-scopus
Metric Benchmark Explanation
Self-citation rate ≥ 200%The journal has a self-citation rate two times higher, or more, when compared to peer journals in its subject field.
Total citation rate ≤ 50%
The journal received half the number of citations, when compared to peer journals in its subject field.
CiteScore ≤ 50%The journal has a CiteScore half or less than the average CiteScore, when compared to peer journals in its subject field.
Number of articles ≤ 50%
The journal produced half, or less, the number of articles, when compared to peer journals in its subject field.
Number of full-text clicks on Scopus.com
≤ 50%The journal's abstract are used half as much, or less, when compared to peer journals in its subject field.
Abstract usage on Scopus.com ≤ 50%
The journal's full text are used half as much, or less, when compared to peer journals in its subject field.
Outlier performance: data science is used to identify journal outlier behavior
Radar tool launched & added to Re-evaluation in 2017: data analytics tool which identifies outlier journals based on 40+ predictors.Examples of prediction behavior:
• Sudden and unexplainable change in article output and growth• Sudden and unexplainable shift in geographical diversity among authors and editors• Shift in received citations and percentage of self-citations
• Flagged journals are manually re-evaluated by the CSAB for (dis)continuation of Scopus coverage.
What is outlier behavior?
0
200
400
600
800
1000
2015 2016 2017 2018
Country 2OtherCountry 1
Big increase in article output after debut in Scopus Shift in geographical diversity
Re-evaluation 2018 results broken down: overall discontinuation rate of 51%
263 Titles re-evaluated
100
38
125
53
47
21
17
61
64
Underperformance
Outlier performance
Publication concerns
53% Discontinuation rate
55% Discontinuation rate
49% Discontinuation rate
Since the launch of the program 772 journals have been re-evaluated with a discontinuation rate of 58%.
Should the CSAB decide a title is not meeting the Scopus Title Selection criteria anymore:• The publisher is informed of the situation by Scopus Re-evaluation team.
• For those journals that are discontinued, no new content is accepted nor backfilled.
• Content already indexed will remain as a matter of scientific record and to ensure the stability and consistency of research trend analytics, a core value element of Scopus.
• In exceptional cases of proven severe unethical publication practice, content already indexed in Scopus may be removed.
Title selection criteria: https://www.elsevier.com/solutions/scopus/how-scopus-works/content/content-policy-and-selection
All journals discontinued in Scopus as part of the re-evaluation program are publicly available via the Scopus discontinued sources list.
The pink elephant in the room and question perhaps on your mind too:
Does Scopus index Predatory journals?
Predatory journals in Scopus paper
• In March 2017 the article Predatory Journals in Scopus, identifying
predatory tiles indexed in Scopus, was published by IDEA, the
Institute for Democracy and Economic Analysis, at the Czech
Academy of Science.
• The paper is based on 2015 Scopus data which is used to analyse
publication behaviour.
• All of the 137 suspicious titles mentioned in the paper have gone
through the re-evaluation process. 97 titles (71%) discontinued.
• All identified titles listed by Beall that are mentioned in the paper
(note that Beall’s List is no longer being maintained) have also gone
through the re-evaluation process. 65% discontinued.
Does Scopus index predatory journals?
• As a leading abstract & citation index, it is not surprising that Scopus has been the subject of an increasing number of questions about predatory content.
• The decision to re-evaluate and potentially exclude a journal takes time and is complex, given the ongoing risk of discontinuing legitimate sources.
• Re-evaluation is a state-of-the-art defence against predatory publishing. Scopus takes its
responsibility in terms of curating its content on an ongoing basis and ensures only the
highest quality content remains indexed in the database.
• Scopus, together with the CSAB, are committed to continue to develop new
approaches to ensure the combined quality and breadth of its research data continue to
be unrivalled and can be trusted for your decisions in research.
Thank youQuestions?
Overall re-evaluation (‘16, ‘17, ‘18) results broken down: overall discontinuation rate of 58%
772 Titles re-evaluated
332
68
367
168
164
42
26
237
130
Metrics
Radar
Publication concerns
51% Discontinuation rate
62% Discontinuation rate
65% Discontinuation rate
Low quality or predatory publishing & Scopus
• When making decisions about science, it is essential that these decisions are based on curated, high-quality data that you can trust. Therefore, low quality or predatory journals are a threat to the integrity of science.
• Because predatory publishing is ill-defined and subject to personal interpretation, independent review of individual journals by academic subject experts in each field is essential.
• Scopus has been rigorously addressing this issue for some years now and have developed a process of continuous monitoring and re-evaluation by the independent Content Selection and Advisory Board (CSAB), to determine whether coverage of the journals in Scopus should be continued or not.
Structural approach: Ongoing content curation to ensure continuous high quality content
Identification of under performing journals
using metrics and benchmarks
Machine learning tool to identify journals with outlier performance
Direct feedback from users and stakeholders
on publication concerns
Re-evaluation by the Content Selection & Advisory Board (CSAB)
Content Curation
Ongoing curation of the full journal base is expected and essential to keep journals at high quality levels and keep the database trustable
Review:
Curate:
Re-evaluation results
772 Titles re-evaluated
332
68
367
168
164
42
26
237
130
Under-performance
Outlier-performance
Publication concerns
51% Discontinuation rate
62% Discontinuation rate
65% Discontinuation rate
Source: Science Magazine. April 3, 2019
Top Related