Dr Rashidi

How to read, write, and evaluate a scientific paper

www.michaelbach.de/ot/

Is Science Publication?Science is– Public– Objective– Predictive– Reproducible– Systematic– Cumulative

Publication makes this possible– Final step in discovery

Is Science Publication?Science– Must be communicated to exist

Medium of communication– Publications: results contribute to scientific

evidence when published – Meetings presentations & abstracts: of only

temporary value

Comparing Scholarly Journal & Popular Magazine Articles

Scholarly Journal Articles Popular Magazine Articles

Written FOR scholars in a specific subject area

Written FOR general public

Written BY an expert or researcher; i.e. an artist, historian, doctor, educator

Written BY a journalist, celebrity, general person

Very little advertising; graphics specialized, if any

Many advertisements and photographs

Reading level is advanced; use specialized vocabulary

Reading level is very basic for general population

Published quarterly or bi-monthly, usually

Published weekly or monthly

Comparing Scholarly Journals & Popular Magazines—cont.

Scholarly Journal Articles Popular Magazine Articles

Include references to sources, bibliographies, full citations (at end of article)

Rarely include references, bibliographies, or citations

Indexed in specialized databases such as ERIC (education articles)

Indexed in general databases such as Reader’s Guide or SIRS

Discuss specific topics in a scholarly subject

Include current events, trends, opinions and popular interest topics

Often include an abstract or brief summary of the article to inform the reader

Often include inviting title and subtitles to encourage reading the article

Approved for publication by a group of specialists in the same field of interest; this is called “peer reviewed”

Approved for publication by the magazine’s editors.

The science of ‘trashing’ a paper

Unimportant issue

Unoriginal

Hypothesis not tested

Different type of study required

Compromised original protocol

Sample size too smallPoor statistics

Unjustified conclusion

Conflict of interest

Badly written

Why should I read?

I n medical field:– To find out whether to use a (new) diagnostic test or

treatment– To learn clinical course and prognosis of disease or

treatment– To determine etiology & causation– To distinguish useful from useless (or harmful)

therapy

You don’t need more information you need the right information

Publish or Perish?

Why shouldn’t I read?John W. Kirklin (pioneer heart surgeon and journal editor)– 5%—no more than 10%—of articles published in

cardiothoracic surgery contribute to new knowledge– Those get lost in the 90% to 95%– Few know how to sort them out– I don’t want to be a patient right after a medical

meeting

A general rule of thumb, regarding what goes where, when both reading and writing a scientific article is

Title: Short, concise, eye-

catchingAbstract:

Why & Why

Introduction:When, Whom,

Why, What & Why

MethodHow & Where Results:

What

Discussion:Why & How

Note: Some journals will allow the Results and Discussion sections to be combined. In this case, the What and the why are presented together.

It’s not a great read!

IMRD structure– Introduction

What question was studied?– Methods

How was the question studied?– Results

What was found?– Discussion

What do the results mean?

IMRD Light

Introduced– 1860s-70s, Pasteur

Neither– Prose nor poetry– Not literature

A roadmap– with standardized signposts

Scientific Publication1658: Journal des Sçavans1812: New England J. of Med. & Surgery1860's-70s: IMRD (Pasteur) format 1957: Published abstract (JAMA)1972: American National Standards Institute: ANSI: IMRD as norm for scientific reporting1978: Uniform requirements for manuscripts submitted to biomedical journals1987: Structured abstract

IMRD HeavyAdditions– Title, authors, affiliations– Various forms of abstract(Descriptive, Informative

and Critical) OR (Structured or Unstructured abstract)

– Subsections– Tables & figures– References– Acknowledgements & disclosures– Appendices– Electronic supplements

Breakdown!

– <20% of readers follow IMRD order– ~50% of editors and 1/3 of reviewers follow IMRD

So why adopt IMRD?– As a standardized aid for selective, strategic

reading

Gasp!

Scientific literature is not widely read?– There is too much to grasp– Paper read thoroughly only by a few writing on

same subject– So: typically scientific papers are scanned—read

selectively and strategically

Selective ReadingTitle – 100%Abstract (first & last lines) – 93%Abstract (the rest) References – 60%Introduction – 40%Methods – 33%Results (especially figures, tables) – 27%Discussion – 27%

Title

What is paper about?

Title

Introduces the workFirst thing read– Usually it is ONLY thing read

Serves to entice intended readersHow do you evaluate a title?Characterize a good title

TitleCharacteristics of good titles– Short, but specific (not an abstract!)– Truly represents content– Might…• Be challenging or controversial• Ask a question• Make statement of conclusion

– IndexableAvoid– Qualifiers, jargon, abbreviations, filler

Title

Evaluation– Does title tell you what paper is about?– Does it exaggerate contents?– Is it too dry to entice readers?– Is it “too cute”?– Does it mislead?

The body of the paper: The whole story

Title: Fishing for readers

Abstract: The “Reader’s Digest” version

A scientific paper is really three (3) separate papers. This fact is critically important when you set out to

write a paper, or to read one.

Although, in published form, the title comes first and the abstract second, they are nearly always the last to be written

AuthorsWho wrote this?

Why important?Like it or not, it is an issue of

authority or expertise or experience Affiliation, H-Index, Co-workers

Conflicts of interest / disclosures

Ultra-Mini Abstract

What is the essence of this study—the “take home” message?

If reader is interested…Robert Day– Clearly stated problem– Clearly stated conclusion

Steven Laureys– Develop a central message and write

everything else to support it

JWK / EHB– Ultramini Abstract: essence of findings for

writer and reader

AbstractFor readers– Scanning tool

For authors (~3 hour’s effort)– Best preparation for writing paper– the roadmap!

Content– 1-3 sentences (~50 words) about the essence of

the study—its message—its inferences

Abstract

Should I read the article?

Abstract

Meeting abstract– Purpose: to get on program

Paper abstract– Summarizes information and data contained in more

complete form in IMRD aspects of manuscript– States conclusions (“bottom line”)– Self contained: Constituting a complete and

independent unit in and of itself– #2 item read (after title)

In fact…

For most readers reading selectively and strategically– Skim first line to understand problem addressed– Skim last line for conclusions

No sense– Concluding by only again summarizing results that have

already been summarized!

AbstractEvaluation– If not structured, read it in structured fashion– Are purposes clearly stated?– Do conclusions match 1:1 the purposes of study– Do methods clearly tell me the study group (e.g.

animals, patients)?– Is there supporting data for each stated purpose &

conclusion?

IntroductionResearch reports in academic journals almost always begin with an introduction in which literature is cited.This integrated introduction and literature review has these five purposes:• introduce the problem area,• establish its importance,• provide an overview of the relevant literature,• show how the current study will advance knowledge in the area, and • describe the researcher’s specific research questions, purposes, or hypotheses, which are usually stated in the last paragraph of the introduction.

Introduction

What is the Problem?Why is it Important?

What is the Approach?

Introduction4 short segments– Problem statement• Does not review field

– Why is it important?– What is context?– Purpose of study• Sets complete roadmap for paper• Slavishly followed in order and with same words for

rest of paper

The IntroductionBegin with the broadest scope and get progressively narrower, leading steadily to the statement of objectives in the last sentence or paragraph of the Introduction.

Introduction or literature or BackgroudEvaluation

Question 1: If there is extensive literature on a topic, has the researcher been selective?

Question 2: Is the literature review critical?Question 3: Is current research cited?Question 4: Has the researcher distinguished between

research, theory, and opinion?Question 5: Overall, is the literature review portion of

the introduction appropriate?

Materials and Methods

How was the study done? Should I believe this study?

Materials & Methods

For selective, strategic readers– Rarely read in entirety if at all– Assumes this section has been vetted by peer

review processFor reviewers– Inadequacies often identified

For science– Is study valid?– Is it replicable?

Materials and Methods

If patients (for example)– What was done?– Where?– Time frame?– Context?– Inclusion/exclusion criteria?– How many (CONSORT [CONsolidated Standards of

Reporting Trials] diagram)?– Characteristics of patients?

CONSORT Flow Diagram

How was study group assembled?– Base group included– Specific exclusions– Analysis group

Materials and Methods

Data analysis– Organize according to purposes of study– Provide detail or references to technical

methodology– BUT don’t leave loopholes!• Most common error is not listing variables considered

in analyses

Materials and Methods

Presentation– Format of summary statistics– Confidence limits & level– Other special features of presentation

Materials and Methods

Evaluation– A checklist is valuable for authors, evaluators, and

readers– CONSORT is one, but journals may have their own– Often contentious• Old methods• Unfamiliar methods• Complex methods

Results

What was found?

Results

Often read selectively and strategically– Figures looked at the most—even though they are

the first thing reviewers suggest eliminating

This is core of paper

Results

What results should be shown?– Selected, well-digested data & findings– Relate directly to purposes of paper, organized

according to purposes, using identical words– No interpretation– No repetition of text, tables, figures

ResultsPart of the truth– Not the whole truth

Themes– Accuracy– Brevity– Clarity

Future– Repository of raw data for reanalysis

ResultsEvaluation– Are data presented that convincingly support

conclusions?• Logical pieces all there• Results stated accurately

– Are there appropriate expressions of uncertainty?– Do negatives reflect underpowered study?– Are methods mixed with results?

Results

Evaluation– Tables• Appropriate• Complete for their purpose• Statistically sound

– Figures• Appropriate information content• Complete legend• Readable

Discussion

So what?Who cares?

Discussion

What do results mean?– Interpretation– Relationships among results– Generalizations– Theoretical implications

Discussion

What do results mean?How do they relate to cumulative knowledge?– Support– Contradict– Completely new

How should I use them?– Practical application

DiscussionSuggested outline– Summarize findings (controversial)– Principal findings• Organized by purpose-driven roadmap• Put results in context of others

– Limitations– Conclusions• inferences• Recommendations

The DiscussionBegin with the interpretation of the data, with respect to the specific objectives of the study, and then get progressively broader, interpreting papers by others, ending with the concepts used to start the Introduction.

DiscussionEvaluation– Is it concise and focused strictly on purposes of

study?– Is interpretation of study reasonable?– Have others been quoted and represented

accurately?– Are inferences supported by results?– Is speculation identified? (Reasoning based on

inconclusive evidence)– Are there promissory notes?– Are new results presented?

References

Can I verify claims and arguments?

ReferencesNot exhaustive– 30 or less is sufficient– Not just recent literature

Contextual– Place subject in context (relevant)– Represents all sides of controversy– Truly relevant

Cited accurately– NLM has a problem!

Template for Taking Notes on Research Articles:

Easy access

for later use

The following 3 reading series were published in core medical journals and contain in-depth

information on critical appraisal and evidence-based practice.From the British Medical Journal (BMJ): How to Read a Paper: The Basics of Evidence-Based MedicineAvailable at: http://www.bmj.com/collections/read.dtl Evidence-Based Nursing NotebookAvailable at: http://ebn.bmj.com/cgi/collection/notebookFrom the Journal of the American Medical Association (JAMA): User Guides to Medical LiteratureAvailable at: http://www.shef.ac.uk/scharr/ir/userg.html

Recommended Reading

Online Critical Appraisal ToolsThe Centre for Methods and Tools (http://www.nccmt.ca/) is currently developing a critical

appraisal framework that is specifically tailored topublic health research. This guide will be the definitive critical appraisal resource for public health

practitioners in Canada.• AGREE InstrumentAvailable at: http://www.agreetrust.org/instrument.htm The purpose of the Appraisal of Guidelines Research & Evaluation (AGREE) Instrumentis to provide a framework for assessing the quality of clinical practice guidelines. Itoriginates from an international collaboration of researchers and policy makers who worktogether to improve the quality and effectiveness of clinical practice guidelines byestablishing a shared framework for their development, reporting and assessment.• A Beginner’s Guide to Judging Research StudiesAvailable at http://www.cihr-irsc.gc.ca/e/34192.htmlThis is a succinct guide to critical appraisal written by John Frank, Scientific Director atthe Canadian Institutes of Health Research. Although this article focuses on makingsense of media reports of new health research, the astute questions Frank poses can beapplied directly to research studies themselves.

Online Critical Appraisal Tools• CASP Critical Appraisal ToolsAvailable at: http://www.phru.nhs.uk/Pages/PHD/resources.htmThe UK-based Critical Appraisal Skills Programme (CASP), a division of the

NHS’s Public Health Resources Unit, has developed a customized critical appraisal checklist for each common type of research study. Direct links to each of the 6 appraisal tools are

listed below: • Qualitative Research http://www.chsrf.ca/kte_docs/casp_qualitative_tool.pdf • Review Articles (including Systematic Reviews) http://www.phru.nhs.uk/Doc_Links/S.Reviews%20Appraisal%20Tool.pdf • Case Control Studieshttp://www.phru.nhs.uk/Doc_Links/Case%20Control%2011%20Questions.pdf

Thanks