Introduction and Framework INLS 541: Information Visualization Brad Hemminger.
Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library...
-
date post
20-Dec-2015 -
Category
Documents
-
view
215 -
download
1
Transcript of Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library...
Scholarly Communications:Changes to Peer Review
• Bradley Hemminger
• School of Information and Library Science
• University of North Carolina at Chapel Hill
Scholarly Communications Process
Idea
V1
Present to colleagues
V2
Present at conference
V3
Submit to journal V4
Referees Revision for journal V5
Journal Final Revision
V6
Revision to update analysis
V7
Revision to include additional new results V8
Scholarly Communications Process
Idea
V1
Present to colleagues
V2
Present at conference
V3
Submit to journal V4
Referees Revision for journal V5
Journal Final Revision
V6
Revision to correct analysis
V7
Revision to include additional new results V8
formulate discussion discussion,revision
Two peer reviews
CopyproofingCriticisms, new thoughts,revision
new results,revision
commentscommentscomments
Author revision
Scholarly Communications Process: What’s Produced
Journal Final Revision
V6
Scholarly Communications Process:What I’d like to see saved!Idea
V1
Present to colleagues
V2
Present at conference
V3
Submit to journal V4
Referees Revision for journal V5
Journal Final Revision
V6
Revision to correct analysis
V7
Revision to include additional new results V8
formulate discussion discussion,revision
Two peer reviews
CopyproofingCriticisms, new thoughts,revision
new results,revision
commentscommentscomments
Author revision
Peer Review Output
Review
(Peer)
With Respect to XYZ…Accept reject revise
Comments to Author
Qualitative Grade
Qualitative Comments
Article
Generalized Review Model
Review (open, peer, machine)
Accept,Reject,Revise,With respect to XYZ
Comments to Author
Qualitative Grade
Quantitative Grades
Score (1-10)
Qualitative Comments
Article
Overview of Peer Review
Review
Peer, Open, Machine
Accept, reject, revise with respect to XYZ standards
Comments to Author
Qualitative Comments
Quantitative Grade
Published Article
Article submitted
Send elsewhere
Filter
Reject
Score (1-10)
Qualitative Grade
General Review Model Parallels
• In general, you have sample (material) which is judged/scored quantitatively and qualitatively by an identified observer.
• Current Peer Review
• Automated Scoring Systems (lab tests)
• Moderated email lists (announce)
• Moderated Eprints servers (arXiv)
Peer Review Options• Human Judgement
– Expert peer review (status quo)– Certified expert peer review– Open Peer Review BMJ, BioMed– Open comment review pyscprints
• Computer Judgement– Computer peer review
• Human Usage– Citation-based (CiteSeer)– Usage counts (CiteSeer) Example– Quantity of discussion
• Coarse Categorization– Two Tier (grey/gold)– Moderator (current arXiv)– No review (old arXiv)
Quantitative
Score (1-10)Score (1-10)Score (1-10)Score (1-10)
Score (1-10)
#citations#hits#number of
related discussions
QualitativeRel Yes/No GroupRel Yes/No GroupAbsolute Absolute
Rel Yes/No GroupRel Yes/No Group
YYYYY
?
Judgment based on some combination of reviews/commentsIdea
V1
Present to colleagues
V2
Present at conference
V3
Submit to journal V4
Referees Revision for journal V5
Journal Final Revision
V6
Revision to correct analysis
V7
Revision to include additional new results V8
formulate discussion discussion,revision
Two peer reviews
CopyproofingCriticisms, new thoughts,revision
new results,revision
commentscommentscomments
Author revision
What areas of improvement?
• Review Process Change
• Search, Retrieval Process Change
• Service Provider Process Change
Review Process Changes
• Include open reviews and comments to get additional feedback.
• Support normal scholarly discourse, allowing give and take with author responding.
• Add quantitative scores to allow better filtering based on quality during retrieval
• Add machine (automated) reviews
Search and Retrieval Changes
• Universal Archive: all material freely available.• Universal Searching: standardized metadata (
Dublin Core) for general searching.• Automated agents to bring material of interest to
your attention.• Use additional review scores (public reviews,
machine) to help filter search.• Example: article scores > 7.0, refereed,
citation count above X, type=research article, search terms = schizophrenia, geneX)
Provider Service Change
• What is worth paying for?– Quality review (Faculty of 1000)– Proofing, citation linking, professional presentation (
CiteSeer, Cite-base)– Archival (JStor)
• Who hosts material:– Society (arXiv)– Commerical Publishers (Elesiever,BioMedCentral)– University Library (MIT Dspace)
New Frameworks for Peer Review
• As an enabling technology: frameworks like NeoRef supports all of the above models in any combination at the same time, while eliminating many of the costs.
• Requirements: Based on OAI and Dublin Core, and expectation of logical universal archive, an universal unique object IDs (URL, DOIs) and person IDs.
Example Model (NeoRef)• All material and metadata are author contributed to a
public OAI archive (author retains ownership).• All materials universally available via search engines that
harvest metadata from OAI archives.• OAI archives have automated or manual moderator to
filter out “junk”.• Everything--articles, reviews, comments, indexings, etc.,
are stored as digital content on archive using the same mechanism. Reviews contain quantitative score, qualitative grade, qualitative comments. Logically (although not physically), a two tier (Grey & Gold) system for materials– High quality keep forever material reviewed by known entity– Grey material (everything else)
NeoRef for Movies, Dates, DocSouth
• The same process used by NeoRef to support Scholarly Communication could be used for almost any purpose. All that is required is storage of Digital Content Items, and linking of reviews, comments, etc to them.
• Movies: Grey is everyone’s reviews; Gold is Siskel and Ebert reviews
• DocSouth: self cataloged and indexed items are Grey; librarian/archivist cataloged and indexed items are Gold.
Can we save the Gold and Grey?
Idea
V1
Present to colleagues
V2
Present at conference
V3
Submit to journal V4
Referees Revision for journal V5
Journal Final Revision
V6
Revision to correct analysis
V7
Revision to include additional new results V8
formulate discussion discussion,revision
Two peer reviews
Author revision
Criticisms, new thoughts,revision
new results,revision
commentscommentscomments
Copyproofing
NeoRef Storage Model
Conference paper (v3)
Comments on V6
Journal Submission V4
Journal Final Revision V6
Revision to include additional results and analyses V8
Auto-indexing
Material expressing content
Two peer reviewsLocal powerpoint Presentation v2
Comments on V3
Automated
Author Indexing
Recognized Expert
Open (anyone)Top Tier (Keep Forever)
Filter (Moderate)
Grey Literature
Author
Machine Review
What do users want?
The Association of Learned and Professional Society Publishers (ALPSP) Survey
Authors and Electronic Publishing• Scholarly research communication has seen far-
reaching developments in recent years. • Most journals are now available online as well as
in print, and numerous electronic-only journals have been launched;
• the Internet opens up new ways for journals to operate.
• Authors have also become conscious of alternative ways to communicate their findings, and much has been written about what they ought to think.
ALPSP felt that it would be timely to discover what they actually thought and what they actually did. This survey aimed to discover the views of academics, both as authors and as readers. Some 14,000 scholars were contacted across all disciplines and all parts of the world, and nearly 9% responded; their detailed comments make thought-provoking reading.
Alma Swan and Sheridan Brown. Authors and Electronic Publishing: The ALPSP Research Study on Authors' and Readers’ Views of Electronic Research Communication. (West Sussex, UK: The Association of Learned
and Professional Society Publishers, 2002).http://www.alpsp.org/pub5.htm
Importance of the Peer Review Process
0102030405060708090
100Peer-reviewed
Refs' commentspublished
Referees identified
Public commentary oneprints
Post-publication publiccommentary
Ability to submitcomments
http://www.alpsp.org/pub5.ppt
Importance of journal features
010
20
30
40
50
60
70
80
90
Citation links Additionaldata
Addit/colourimages
Manipulablecontent
Video/sound
Importance of the peer review process
0102030405060708090
100Peer-reviewed
Refs' commentspublished
Referees identified
Public commentary oneprints
Post-publication publiccommentary
Ability to submitcomments
Importance of publishers’ rolesFactor Responses as authors Responses as readers
Peer review 81 80
Gathering articles together to enable browsing of content
64 49
Selection of relevant and quality-controlled content
71 54
Content editing and improvement of articles
60 39
Language or copy editing 50 34
Checking of citations/adding links
46 28
Marketing (maximising visibility of journal)
44 20
Importance of future dissemination channels
Dissemination method Very important plus important
categories
Ranking
Traditional print + electronic journal 91 1
Discipline-based electronic reprint archive 78 2
Traditional print journal 77 3
Traditional electronic-only journal 66 4
Institution-based electronic reprint archive 60 5
New forms of electronic-only journal 49 6
Discipline-based electronic preprint archive
44 7
Institution-based electronic preprint archive
33 8
http://www.update-software.com/Cochrane/MR000016.pdf
Cochrane Methodology Review
• Despite its widespread use and costs, little hard evidence exists that peer review improves the quality of published biomedical research.
• There had never even been any consensus on its aims and that it would be more appropriate to refer to it as ‘competitive review’.
Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,”BMJ 326 (February 1, 2003): 241
http://bmj.com/cgi/reprint/326/7383/241/a.pdf
Cochrane Methodology Review• On the basis of the current evidence, ‘the
practice of peer review is based on faith in its effects, rather than on facts,' state the authors, who call for large, government funded research programmes to test the effectiveness of the [classic peer review] system and investigate possible
alternatives. Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,”
BMJ 326 (February 1, 2003): 241http://bmj.com/cgi/reprint/326/7383/241/a.pdf
Cochrane Methodology Review• The use of peer-review is usually
assumed to raise the quality of the end-product (i.e. the journal or scientific meeting) and to provide a mechanism for rational, fair and objective decision-making. However, these assumptions have rarely been tested.
Tom O. Jefferson, Phil Alderson, Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of
Biomedical Studies. (Middle Way, Oxford:Update Software Ltd, 2003).
http://www.update-software.com/Cochrane/MR000016.pdf
Cochrane Methodology Review• The available research has not clearly
identified or assessed the impact of peer-review on the more important outcomes (importance, usefulness, relevance, and quality of published reports)
• … [G]iven the widespread use of peer-review and its importance, it is surprising that so little is known of its effects
Tom O. Jefferson, Phil Alderson,Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of
Biomedical Studies. (Middle Way, Oxford:Update Software Ltd, 2003).
http://www.update-software.com/Cochrane/MR000016.pdf
FURTHERMORE …• 16% said that the referees would no
longer be anonymous• 27% said that traditional peer review
would be supplemented by post-publication commentary
• 45% expected to see some changes in the peer-review system within the next five years
Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15 no. 4 (October 2002): 247-258.
Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdf