How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this...
-
Upload
aldous-pitts -
Category
Documents
-
view
225 -
download
0
description
Transcript of How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this...
![Page 1: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/1.jpg)
How to Referee aTechnical Paper
Saul GreenbergUniversity of Calgary
should I referee?is this paper any good?
![Page 2: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/2.jpg)
The Message
Refereeing is excellent practice for developing critical appraisal skills understanding how good (and bad) papers
are written
![Page 3: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/3.jpg)
Many articles on refereeing– Parberry: A Guide for New Referee in
Theoretical Computer Science– Forscher: Rules for Referees– Guidelines that accompany referee requests
![Page 4: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/4.jpg)
Refereeing
Purpose of Refereeing– quality control
• eliminate bad papers
– choose best papers from a good set
• competition for space
Submissions295 papers
Proceedings(33 papers)
![Page 5: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/5.jpg)
Referee Process a peer review process
![Page 6: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/6.jpg)
![Page 7: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/7.jpg)
People involved. Author
– correctness of argumentation and results – sound grounding in the literature– good quality of presentation– appropriate for the intended audience/venue
![Page 8: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/8.jpg)
People Involved Editor
– whether paper should go out for review– choosing appropriate referees– acceptance/rejection decision– explanation letter
![Page 9: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/9.jpg)
People Involved Editorial support
– good record keeping (copies of paper, reports)– tracking and distributing referee reports,
reminding referees
![Page 10: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/10.jpg)
People involved Referee
– critical appraisal of the paper’s contents– opinion, rationale, changes, suggested action – usually 3 referees/paper
A paperto referee
![Page 11: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/11.jpg)
Referee types– topic specialists
• is/has worked on similar problem• knows literature, other work very well• understands methodologies• considers nuances of your work/contribution
– area specialists• knows general area, and how topic fits within it• considers contribution of work to the general area• evaluates comprehensibility by non-specialists
![Page 12: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/12.jpg)
Why Referee? Fairness
– all your papers will be refereed– expected duty of all researchers/academics
![Page 13: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/13.jpg)
Why Referee? Good practice !! Other upsides
– enhance reputation – expedites processing of your own papers – get on editorial board or program committee – 'previews' to the state of the art
Downside– more work!
![Page 14: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/14.jpg)
Quality Control: Research... Correctness
– of argument/method/algorithm/mathematics/proof... (is a fix necessary?)
Significance – does it work on a valid problem?– will these results make a difference?– is it significant to area/journal, etc...? – will it stimulate further work in the area?– is it more than an obvious/trivial solution?
![Page 15: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/15.jpg)
Quality Control: ...Research Innovation
– original, creative, novel, inventive – not trivial extensions, or combination of old
work with no added value Interesting
– well motivated – relevant (when & where & to whom)
![Page 16: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/16.jpg)
Quality Control: ...Research Replication
– can the work be reproduced from the description by an experienced person in the area?
Timeliness – of current interest to the community – but account for:
• publication delay
![Page 17: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/17.jpg)
Quality Control: ...Research Previous publications
– by other authors– by this author
• www publication, • minor conferences• minor variations of the theme...
![Page 18: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/18.jpg)
Quality Control: Writing Succinct
– message and arguments should be • clear, compelling, to the point
not • hand-waving • obscure/hidden behind jargon, etc.
![Page 19: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/19.jpg)
Quality Control: …Writing Accessible
– is it appropriate to the audience? • specialists & range of generalists • is there something for both?
Language & organization– readable, good grammar/structure reflects care
• people do not have the time to read badly written papers
![Page 20: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/20.jpg)
Quality Control: …Writing Use of figures/tables
– supports the story Title & abstract
– indicates content, summarizes main points English as a 2nd language?
![Page 21: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/21.jpg)
Ethics: Professionalism Act in the best interest of the author &
paper
![Page 22: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/22.jpg)
Ethics: ...Professionalism Constructive critique
– if acceptable, explain how it can be improved – if paper is unacceptable, explain why & where, but
politely Specific rather than vague criticism, e.g.
• 'what' is wrong with the algorithm, rather than 'the algorithm is wrong'
• what related work is missing • key examples of numerous errors
![Page 23: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/23.jpg)
Ethics : Speed Speed
– fast turnover – you are on a critical path! – affects timeliness & publication delays – turnover times:
• conferences: deadlines • journals: approx. 3-6 weeks
![Page 24: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/24.jpg)
Ethics : Objectivity Fairness
– author may use point of view/methodology/arguments different from your own
– judge from their school of thought– remove personal prejudice
• e.g. field, institution, author, nationality, author, association (colleague, friend, rival)
Conflict of interest – discuss with editor – if you cannot be objective, return the paper
![Page 25: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/25.jpg)
Ethics : Confidentiality Do not circulate submitted papers
– except for other reviews/comments (publication-dependent)
Never use/discuss results – but can ask for permission from the authors
Protecting your identity– anonymous reviewing the norm– you may reveal your identity if you wish...
![Page 26: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/26.jpg)
Ethics: Honesty & Courtesy Honesty
– judge your own expertise, – give your own confidence in your appraisal
Courtesy – constructive criticism – non-inflammatory language – no put-downs
I was rejected
![Page 27: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/27.jpg)
Ethics: Dilemmas How many papers to submit/referee?
– 1 paper submitted -> 3 referees (minimum)
How much time should I spend reviewing – enough to give fair treatment – don't rush, the author deserves a fair hearing
![Page 28: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/28.jpg)
Ethics:…Dilemas What if a similar paper has been published?
– journal papers can be reasonable expansions of conference papers
– can be republished if obscure (eg, workshop)
What if I am working on the same problem? – be honest & open -> consult with the editor – be aware of the race for independent co-discovery
![Page 29: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/29.jpg)
A Template for Reviewing
Paper Title Author(s) Manuscript Number
![Page 30: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/30.jpg)
A Template for Reviewing Briefly summarize the paper (2-3 lines)
– can you extract a main message from your paper?
• lets author know if you understood the main message
– “If you can’t, there is probably something wrong with the paper”
– --- CHI FAQ
![Page 31: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/31.jpg)
A Template for Reviewing What is NEW and SIGNIFICANT in the work
reported?– New:
• has it been done before?• is it a rehash / republication of old stuff (yours or others)?
– Significance• in five years time, would the work have an identifiable
impact? (rare)
![Page 32: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/32.jpg)
A Template for Reviewing ...What is NEW and SIGNIFICANT
– Survey/discussion piece• does it add value?
– Would it stimulate further work in this area?• is it a reasonable increment that keeps the research area going
(frequent)?• does it have innovations?• is it interesting?• is it timely to the community?
![Page 33: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/33.jpg)
A Template for Reviewing
Describe the QUALITY of the RESEARCH
– How sound is the work?• quality of algorithms, analyses, evaluation methods, etc.
– How appropriate/reliable are the methods used?• are they adequate to support the conclusions• is it correct?
![Page 34: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/34.jpg)
A Template for Reviewing Describe the QUALITY of the RESEARCH
– How reasonable are the interpretations?• good arguments• alternative interpretations explored/left out
– How does it relate to existing work?• bibliographies, background, important omissions…
![Page 35: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/35.jpg)
A Template for Reviewing Describe the QUALITY of the RESEARCH
– Can an experienced practitioner in the field duplicate the results from the paper and the references?
• are there details sufficient?
![Page 36: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/36.jpg)
A Template for Reviewing Describe the QUALITY of the WRITING
– is the message clear?– is the paper easy to follow and understand?– is its style exciting or boring?– is it well organized?– is there a good flow of logic/argumentation?
![Page 37: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/37.jpg)
A Template for Reviewing Describe the QUALITY of the WRITING
– is it grammatically correct?– are figures and tables used well and integrated
into the text?– if it is a foreign writer, how can it be improved?
![Page 38: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/38.jpg)
A Template for Reviewing How RELEVANT is the work to the
expected readers?– domain – depth of treatment– degree of specialization– accessible to expected range of expertise of
readership
![Page 39: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/39.jpg)
A Template for Reviewing Provide any OTHER COMMENTS you
believe would be useful to the author – constructive suggestions on repairing problems– pointers to missing / relevant work– minor typos/flaws
![Page 40: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/40.jpg)
A Template for Reviewing Provide any OTHER COMMENTS
– If revisions were possible, what should the author do to make this paper publishable?
• concrete, very specific suggestions on what – must be done– optional work
![Page 41: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/41.jpg)
A Template for Reviewing. Rate the papers ACCEPTABILITY
andsummarize why you gave this rating Conference:
Definitely rejectProbably rejectCould go either way Note: equivalence class!Probably acceptDefinitely accept
![Page 42: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/42.jpg)
A Template for Reviewing Rate the papers ACCEPTABILITY
– Journal:Definitely reject Major revisions
– additional work, major reworking of arguments– subject to a careful check by editor/reviewers
Minor revisions– typos, minor changes
Accept as is
![Page 43: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/43.jpg)
A Template for Reviewing Using the scale 1 = Know virtually nothing about this area 2 = Not too knowledgeable, but I know a bit 3 = Know a moderate amount, about average 4 = Not my main area of work, but I know a lot about it
5 = My area of work, I know it well rate your EXPERTISE in the area addressed by
the paper
![Page 44: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/44.jpg)
A Template for Reviewing Provide comments that you feel are relevant to
the review process but that you do NOT want forwarded to the author(s) – try to avoid using this– conflict of interests– pointers to things that would reveal identity– harsher things that would be mis-interpreted– suggestions on how to phrase acceptance/rejection
letters...
![Page 45: How to Referee a Technical Paper Saul Greenberg University of Calgary should I referee? is this paper any good?](https://reader035.fdocuments.in/reader035/viewer/2022062223/5a4d1b297f8b9ab059998813/html5/thumbnails/45.jpg)
Summary
Refereeing is excellent practice for
developing critical appraisal skills– templates, typical flaws
understanding how good (and bad) papers are written– apply understanding to your own work