COMMENTS ON COCHRANE REVIEWS Approaches to managing feedback WORKSHOP 22 September 2013 Cochrane...
-
Upload
lilian-criddle -
Category
Documents
-
view
217 -
download
0
Transcript of COMMENTS ON COCHRANE REVIEWS Approaches to managing feedback WORKSHOP 22 September 2013 Cochrane...
COMMENTS ON COCHRANE REVIEWSApproaches to managing feedback
WORKSHOP
22 September 2013
Cochrane Colloquium, Quebec City
Introductions
We are:• John Hilton Editor, Cochrane Editorial Unit
• Gavin Stewart Associate Editor, Wiley
• Jo Garner Senior Business Analyst, Wiley
• Toby Lasserson Senior Editor, Cochrane Editorial UnitFeedback Editor, Airways Group + 19 other roles
Disclosures:
Management of comments is part our jobs as employees of our respective organisations but otherwise we have no actual or potential conflict of interest in relation to this presentation.
Who are you?
Introductions
We are:• John Hilton Editor, Cochrane Editorial Unit
• Gavin Stewart Associate Editor, Wiley
• Jo Garner Senior Business Analyst, Wiley
• Toby Lasserson Senior Editor, Cochrane Editorial UnitFeedback Editor, Airways Group + 19 other roles
Disclosures:
Management of comments is part our jobs as employees of our respective organisations but otherwise we have no actual or potential conflict of interest in relation to this presentation.
Who are you?
Introductions
We are:• John Hilton Editor, Cochrane Editorial Unit
• Gavin Stewart Associate Editor, Wiley
• Jo Garner Senior Business Analyst, Wiley
• Liz Dooley Managing Editor, ARI GroupME Support + 21 other roles
Disclosures:
Management of comments is part our jobs as employees of our respective organisations but otherwise we have no actual or potential conflict of interest in relation to this presentation.
Who are you?
Workshop• Part 1: What’s the latest?
• Overview, data, guidance, support• What do you need? What do you need to know?
• Part 2: Case studies• Some real-life examples• What would you do? And why? Any barriers, hurdles or brick walls?
• Part 3: The way ahead• Improving the process and the display• How can you help?
• Part 4: Is it worth it?• Isn’t everyone using Twitter these days?• What do you think?
Submitting comments
Submitting comments
Submitting comments
Submitting comments
Comments are also submitted via other channels:
• Direct contact to CRG
• From OVID
• Direct contact to Editor in Chief or CEU
• Summaries website (not at the moment)
Processing comments
Comment submission to Wiley &
CEU
Comment triage by
Wiley
Comment assessment by ME & FE
Comment management
What does CEU do?
• Comments go to Editor in Chief, and two editors• EiC looking if intervention required (with CRG)• Editors looking for cross-CRG issues (e.g. multiple submissions)• Support• Tracking
Processing comments
Comment submission to Wiley &
CEU
Comment triage by
Wiley
Comment assessment by ME & FE
Comment management
What gets filtered out:
• Comments or enquiries about the website functionality, access, display, etc
• Comments that are offensive, nonsensical, or inconsequential• Comments that arise from system errors or system testing• Duplicates
Processing comments
Comment submission to Wiley &
CEU
Comment triage by
Wiley
Comment assessment by ME & FE
Comment management
CRG-led process
• Managing Editor• Feedback Editor• Co-ordinating Editor(s)• Feedback contributor(s)
Processing comments
Comment submission to Wiley &
CEU
Comment triage by
Wiley
Comment assessment by ME & FE
Comment management
CRG-led process
• Managing Editor• Feedback Editor• Co-ordinating Editor(s)• Feedback contributor(s)• Author(s)• Editor(s)
Processing comments
Comment submission to Wiley &
CEU
Comment triage by
Wiley
Comment assessment by ME & FE
Comment management
CRG-led process
• Managing Editor• Feedback Editor• Co-ordinating Editor(s)• Feedback contributor(s)• Author(s)• Editor(s)• Editor-in-Chief• CEU Editor(s)• Steering Group, CEO• Interfering third parties
Acting on commentsQuestions, questions, questions
• Is it in English?• Is it coherent and not clearly wrong?• Is it about the content of the review?• Should it be published?• How much editing does it need?• Are the authors going to respond? And when?• If authors won’t or can’t respond, what then?• How long will the whole process take?• Is anyone in editorial base conflicted?• Has contributor declared conflicts?• Does it trigger an amendment?• Does it trigger an update?• Does it raise broader issues?
Acting on commentsQuestions, questions, questions
• Is it in English?• Is it coherent and not clearly wrong?• Is it about the content of the review?• Should it be published?• How much editing does it need?• Are the authors going to respond? And when?• If authors won’t or can’t respond, what then?• How long will the whole process take?• Is anyone in editorial base conflicted?• Has contributor declared conflicts?• Does it trigger an amendment?• Does it trigger an update?• Does it raise broader issues?
What have I missed?
Process
Judgement call?
Workflow?
Comment outcomes1. Validity
1. Valid: take action
2. Not valid: take no further action
2. Usefulness1. Translate
2. Edit
3. Drastically cut or ask for resubmission
3. Suitability for publication1. Publish
2. Don’t publish
4. Author response1. Not needed
2. Needed but not forthcoming
3. Received and published
5. Action on review1. No action needed
2. Review amendment
3. Review update
4. Changes at next updateC
om
mu
nic
atio
n
How many comments?
2011 Q1
2011 Q2
2011 Q3
2011 Q4
2012 Q1
2012 Q2
2012 Q3
2012 Q4
2013 Q1
2013 Q2
0
5
10
15
20
25
30
35
40
45
Which groups?• In the first half of 2013…
Which groups?• In the first half of 2013…
4 Anaesthesia Group
Heart Group
Pregnancy & Childbirth Group
Wounds Group
Which groups?• In the first half of 2013…
4 Anaesthesia Group
Heart Group
Pregnancy & Childbirth Group
Wounds Group
6 Airways Group
Which groups?• In the first half of 2013…
4 Anaesthesia Group
Heart Group
Pregnancy & Childbirth Group
Wounds Group
6 Airways Group
8 Acute Respiratory Infections Group
Which groups?• In the first half of 2013…
4 Anaesthesia Group
Heart Group
Pregnancy & Childbirth Group
Wounds
6 Airways Group
8 Acute Respiratory Infections Group
And 32 other CRGs received at least 1 comment
Characteristics of comments• Study inclusion/exclusion• Interpretation of findings• Conflict of interest• Scope• Methods• Typos, errors
Characteristics of comments• Study inclusion/exclusion• Interpretation of findings• Conflict of interest• Scope• Methods• Typos, errors
One sentence10 pages
Detailed, well-written referenced letterSemi-coherent scribble
Focused on single aspect of review Multiple issues, not all related to review
Characteristics of comments• Study inclusion/exclusion• Interpretation of findings• Conflict of interest• Scope• Methods• Typos, errors
One sentence10 pages
Detailed, well-written referenced letterSemi-coherent scribble
Focused on single aspect of review Multiple issues, not all related to review
• Retired academics• Cochranites• Students• Clinicians• Pharmaceutical companies or manufacturers• Persistent single-issue enthusiasts
Where do comments come from?
What about the unapproved comments?
Example 1
Review
Speed cameras for the prevention of road traffic injuries and deaths
ContributorMaller
Comment
When there is accident happened, who will take responsibility? After several day or long time, how to know the scene when it happened? Car video recorder could record accident and store video data with gps information on memory card, people could get accident reconstruction, it is very easy. more view: http://www.cheapcarcamera.com
What about the unapproved comments?
Example 3
Review
Probiotics for the prevention of Clostridium difficile‐associated diarrhea in adults and children
ContributorRachel Bock
Comment
Test comment - please delete
Known unknowns?• Who is submitting comments: students, clinicians,
Cochranites, pharma companies? (and why?)• How many comments are published, and how long after
submission?• How many reviews amended or updated as a result of
feedback?• What aspects are commentors focusing on? (study
inclusion, interpretation, outcomes, etc?)• Any more? What would it be useful to know?
• ANY VOLUNTEERS?
Cochrane Editorial & Publishing Policy Resource
http://www.cochrane.org/editorial-and-publishing-policy-resource
Cochrane Editorial & Publishing Policy Resource
http://www.cochrane.org/editorial-and-publishing-policy-resource
Cochrane Editorial & Publishing Policy Resource
http://www.cochrane.org/editorial-and-publishing-policy-resource
Cochrane Organisational Policy Manual
http://www.cochrane.org/organisational-policy-manual
Cochrane Handbook
http://handbook.cochrane.org
Cochrane Editorial Unit website
http://www.editorial-unit.cochrane.org/cochrane-library-feedback
So what’s in those new guidelines?• Guidance on how feedback should be managed• When to publish, when not to publish• Guidance on reasonable timescales• Roles of ME and FE• Meeting expectations of contributors• Where to get support• What to do with non-English comments
Your feedback will be very welcome.
Feedback editors: neglected?
• Communication to and between FEs has been lacking• Need Role description (purpose, expectations)• Induction package?• Page on Cochrane Training page?• Independence of role• Cochrane.org feature• Forum
So what else do you/we need?
Case studies• 1. • 2.• 3. • 4.
What’s in that CEU proposal?• Use the term ‘comments’• Decouple comments from review, but link to version• Display comments parallel to review version• Make comments citable (DOI)• Expectations about reasonable turnaround time• Transparency and tracking• Archie workflow for comments
So that was a while ago. What happened?....
Cochrane-Wiley Roadmap
• A set of work streams fulfilling agreed contractual projects
• Next 18 months• Feedback scheduled for Q2 2014• Project will involve:
• CEU, Wiley, Cochrane Tech team• CRGs• You?
Goals for feedback project
• Make process more efficient and transparent• Phased approach to first improve the user experience for:
• the Contributor• the Reader of feedback
• De-couple publishing feedback from the review or protocol• Make the individual comment citable in its own right
Requirements for user feedback
1. A Contributor is able to submit a comment2. The system shall save and acknowledge the comment3. Wiley Editorial may moderate the comment 4. The system shall trigger an editorial workflow that may result
in publication of the comment along with a response5. A Reader will be able to view comments and responses6. A Reader will be able to cite the comment7. A Search User will be able to search comments and
responses
Proposed approach – submission
Proposed approach – submission
Proposed approach – display
Improve processing
• A system for moderation• Separate the publishing of feedback from the review, retain a
link between them:• Feedback as a new Content type?• Changes to Archie, through XDPS and to Online Library
• Add the ability to cite individual comments & author responses• Improve the robustness of the feedback-response loop
• Feedback items having their own workflow?
Why all the bother?
• So we can “…amend reviews in the light of new evidence…to reflect the emergence of new data, valid feedback, solicited or unsolicited, from whatever source”
• Chalmers I, Haynes B. Reporting, updating, and correcting systematic reviews of the effects of health care. BMJ 1994;209:862-5.
But where are all the comments?
Not just Cochrane
http://jasonpriem.org/2011/01/has-journal-article-commenting-failed/
It can work…
Even then, will authors engage?Peter Gøtzsche, Andreas Lundh and BMJ editors looked at BMJ rapid responses
Even when comments could “invalidate research or reduce reliability” over half the time authors couldn’t be bothered to respond.
http://www.bmj.com/content/341/bmj.c3926
Are they using the ‘shadow system’ - scribbled marginalia, chats in labs, peer review, discussions at conferences, journal clubs
And Twitter, of course.
So, why no comments?“Why is it that comment forums provided by online scholarly journals for post-publication peer review are so consistently underutilized, even though online forums concerning countless subjects, from culture to politics, are booming?”
• Technophobia? Time?• Worry about appearing to be ‘unacademic’• “Commenting on scientific articles does not advance my career”• Prefer a published letter (despite delays)• Disincentives: fear of upsetting, being wrong, giving away ideas
Is the question not whether have comments failed, but whether are they succeeding somewhere else?
What’s different about Cochrane?• Updating! Comments are part of a continuous process of
improvement and updating• Culture of acceptance of criticism• Expectations of authors
So what could we or you do?
• Proactive solicitation• More encouragement on website – expose comments,
prompt for comments• Altmetrics• Activate specific social networks and establish rapport