The Basics of Reading A Research Article

Amy Gutman MD ~ EMS Medical Directorprehospitalmd@gmail.com / www.TEAEMS.com

OBJECTIVESInformation Presented Meets or Exceeds NREMT Educational Standards

EMS Research History

Definitions

Breaking down research articles

Evidence-based medicine

Research resources

Summary

OVERVIEW

Much “sensationalized” but bad research Cellphones cause brain cancer

Vaccines cause autism

How do you tell “good” from “bad”? Is “good” research “important”?

Review basics of reading a research paper so you can keep up with current trends in prehospital care

WHAT IS RESEARCH?

Science is systematized knowledge derived from observation & experimentation

Research is a systematic investigation to discover or establish facts by utilizing the “scientific method”

The scientific method is how researchers construct an accurate, reliable & repeatable answer to a question

SCIENTIFIC METHOD Anyone can perform &/or read research

Observe something

Ask a question, or “hypothesis”, about observation

Use hypothesis to make prediction

Test prediction by experimentation

Modify hypothesis after reviewing experimentation results

Repeat until satisfied hypothesis answers question

Ask more questions!

HYPOTHESIS & PREDICTIONS

A hypothesis is a question or theory based upon an observation

Often common sense, but common sense is neither common, nor true

Examples: “Why do I get long-distance transports at

the end of my shift?”

“Does anyone really drink ‘two beers’”?

SCIENTIFIC METHOD: EXAMPLE

Observation, Hypothesis & Prediction “My Medical Director is a an awesome, short, chunky,

red-headed female. Therefore I believe that ALL awesome Medical Directors are short, chunky, red-headed females”

Testing the Hypothesis / Methods: “According to providers I asked, there are at least two awesome Cape Cod Medical

Directors who are not red-headed, short, chunky or female”

Results: “Because there are awesome Medical Directors who didn’t meet the hypothesis, I

modified the hypothesis to “One awesome Medical Director is short, chunky & red-headed, but not all awesome Medical Directors must be physically similar in appearance”

Reproducibility & Clinical Applicability: Easily be reproduced in any prehospital environment Medical Director physical characteristics may not indicate awesomeness

EMS RESEARCH

The goal of EMS research is integrating results into clinical practice after an evidence-based initiative to improve patient outcomes in a competitive & cost-conscious healthcare market

• Domains: Clinical: Study of direct patient care

Systems: Effects of EMS system design & operations on resource utilization

Educational: Training methodology for prehospital providers

EMS RESEARCH HISTORY

EMS practices initially modeled after battlefield/hospital care 1st paper published in 1966 (BMJ) showed patients “suffocated” due to poor EMS training

1973 EMS Systems Act (Public Law 93-154) funded development of regional EMS systems, protocols & research Pre-1980 only 3 “Efficacy Studies” published that evaluated if a practice or drug works in

ideal conditions (but not necessarily in the field!) A 1989 editorial stated “Impressively Deficient” data for “efficacy of scope of EMS practice”

1993 NHTSA FEMSC “EMS for Children Report” identified need for pediatric research

2001 NHTSA National EMS Database founded to answer research questions 1st paper evaluated prehospital ETI in pediatric cardiac arrest & found that most providers

lacked training & that patients worse outcomes

Today: better funding, peer-reviewed journals & some clinically relevant research

EMS RESEARCH BARRIERS

Few trained researchers with minimal funding

Failure to translate research into clinical practice

Lack of integrated IT systems linking prehospital data with patient outcomes Multiple locations / personnel / researchers

HIPPA, IRBs

Unstable patients often w/o diagnosis at time research conducted Uncontrolled research environment

Critical nature limits practical & ethical experimental interventions

Difficulty in obtaining informed consent

CURRENT NATIONAL EMS RESEARCH (www.EMS.GOV/Research)

National EMS Assessment (2009-2011) analyzed 20,000 agencies, 82,000 vehicles, 826,000 personnel

EMS Evidence-Based Guidelines

National EMS Information System (NEMSIS)

Longitudinal EMS Demographic Study

National EMS Research Agenda

National EMS Assessment

EMS Workforce for the 21st Century: Feasibility for Safety & Surveillance

EMS Systems Configuration Study

EMS Performance Measures & Outcomes Evaluation

Motor Vehicle Occupant Safety Survey: Crash Injury & EMS

EMS RESEARCH ORGANIZATIONS / WEBSITES

PREHOSPITAL RESEARCH JOURNALS JEMS is most widely-read print journal

Not peer-reviewed

Opinions, summaries & commentaries that are informative but not necessarily evidence-based

Peer-Reviewed Journals Prehospital Emergency Care

Prehospital & Disaster Medicine

Annals of Emergency Medicine

Pediatric Emergency Care

Air Medical Journal

EVIDENCE-BASED MEDICINE (EBM)

Scientific method to answer a question & make patient care decisions

Establish “standards of care” after multiple studies replicate similar results, or a multi-center study shows undeniable results Example: Maine Selective Spinal Clearance

Study

In cost-cutting age, insurance often will not reimburse for non-EBM practices Litigation possible if non-EBM procedures

PEER-REVIEWED RESEARCH

Rigorous review by persons of similar credentials to researchers

Maintains standards of quality & provides credibility by critically evaluating research & exposing flaws prior to publication

A review will find the article: Acceptable (rare)

Acceptable but requires revisions (common)

Not acceptable (common)

HOW AN ARTICLE GETS TO PRINT (TIMELINE)

Days 1-7 Hypothesis Formulated

Days 1-60 Methodology formulated; Funding application begins

Days 30-120 IRB proposal written, submitted, resubmitted

Months 4-5 Identify & recruit investigators & subjects

Month 6-9 Investigator training; Research,data collection & article prep begins (background, methodology)

Months 9-? For RCTs 12-24 months, Chart Reviews usually 3-6 months

Year 2+ Abstract prepared & submitted for conference presentation; article prepared for submission

Year 2+ 8 Mo If accepted for revisions, editing begins. If refused, hunt for another

journal begins

Year 2+ 10 Mo Article accepted; published 3-6 mos later unless special

circumstances

• Peer-reviewed EBM articles appear in many journals (some prestigious, some not)• Most research happens years before publication

JUDGING A RESEARCH PAPER Credible source?

Applicable to EMS?

Design appropriate to answer question being asked?

Variables controlled to ensure reliability of results?

Results conclusive / logical?

Alternative explanations for results?

Do findings impact current evidence or require changes in clinical practice?

How biased is the research?

BASIC VS APPLIED RESEARCHBASIC APPLIED / CLINICAL

Performed in lab or clinical setting as preliminary data collection to refine a hypothesis

Independent variables manipulated to observe & describe effect(s) on a dependent variable

Often examines properties of drugs or used in development of diagnostic tests

Every drug starts as a chemical whose properties are examined through experimentation

Real subjects in real-world situations

Effects of a drug, device, or procedure studied on humans

Test & prove (or disprove) therapies, procedures & protocols

LEVELS OF SCIENTIFIC EVIDENCE

Ia: Randomized Controlled Trial (RCT) meta-analysis

Ib: RCT (single)

IIa: Prospective Non-RCT

IIb: Experimental study (case series, animal studies)

III: Retrospective or Descriptive studies (case-control, comparative,

correlation studies)

IV: Expert opinions, extrapolations

V: Rational conjecture

Strongest

Weakest

RANDOMIZED CONTROLLED TRIAL (RCT)

Subjects randomized into control & experimental groups

Groups evaluating for variables & outcomes of interest Example: Outcomes of STEMI patients

given aspirin vs those not given aspirin

Uses the scientific method & most “valid” research method

RCT EXAMPLE Mattox KL, Bickell B, Pepe PE*, Burch J, Feliciano D.

Prospective MAST study in 911 patients. J. Trauma. 1989;29:1104-12

911 adult trauma pts with SBP <90mmHg randomized by alternate days & transported to Ben Taub Experimental Variable: MAST or no-MAST

Dependent Variable: survival from trauma

Independent Variables: etiology, age, race, sex, location of injury, trauma scores, injury severity scores

Results: Mortality 31% MAST group, 25% in non-MAST group

Conclusions: MAST trousers in trauma pts increases mortality

RCT: COHORT STUDIES

Subjects with a condition or receiving a treatment compared with another group not affected by condition / treatment under investigation

Different than a double blind study as this is an observational study only No intervention by the researchers

EXAMPLE: COHORT STUDY

Gottesman BE, Gutman A et al. Radiation exposure in emergency physicians working in an urban ED: a prospective cohort study*.

HYPOTHESIS: EM MDs are exposed to radiation levels above NCRP limits

METHODS: Prospective cohort study conducted at a level I Trauma ED. Dosimeter radiation badges placed on all ED MDs & their phones carried 24/7/365. At the end of the study exposure dose for each subject calculated

RESULTS: 75 MDs enrolled. Compliance 99%. Annual extrapolated exposure for subject with highest radiation level was 50 mrem, below 5000 mrem exposure limit for health care workers

CONCLUSION: EM MDs working in an urban, academic, level I trauma center ED are not at risk of exceeding NCRP dose limits for ionizing radiation exposure

*Am J Emerg Med. 2010 Nov;28(9):1037-40

RCT: DOUBLE BLIND STUDY

Neither subject(s) nor researchers knows whether subject receiving treatment, or control (placebo) treatment

Example: Aufderheide TP, et al. Resuscitation Outcomes Consortium (ROC) PRIMED trial:

rationale & methodology for the impedance threshold device (ITD)* AIM: Compare OOHCA survival to discharge between CPR+ITD vs CPR vs fake ITD METHODS: Prospective, double-blind, RCT in OOHCA pts by EMS systems

participating in the ROC. 1.4% absolute survival difference found in CPR + ITD CONCLUSION: If ITD demonstrates hypothesized survival improvement, an

estimated 2700 deaths from OOHCA could be prevented annually*Resuscitation. 2008 Aug;78(2):179-85

SYSTEMATIC REVIEWS (RCT META-ANALYSIS)

Summary of literature & statistical analysis after detailed search to determine effectiveness of interventions / procedures

Results of multiple studies evaluating a similar question are pooled & statistically analyzed

Highest level of scientific evidence as reduces possibility of false negative results Poor quality studies are often excluded from

the statistical analysis

Because patient numbers are so high (hundreds to thousands), minimal chance of inaccurate conclusion by “chance”

ANIMAL RESEARCH

Precursor to evaluating a human intervention

Computer models often initially used vs animal models for ethical reasons

Results often cannot be extrapolated to draw conclusions on what will happen in humans Isoproterenol dosing based on rat studies resulted in 3500 British asthmatics

dying from overdoses Pharma, 1971; vol 18:272

Domperidone withdrawn from market after 25 pts died from lethal arrhythmias not seen in dogs with 70 X toxic human dose Drugs, 1982, vol 24:360-4

Digitalis testing delayed in animals, while used safely for decades in humans because of toxic rodent effects Tox in vitro, 1992, vol 6:47-52

EPIDEMIOLOGICAL / CORRELATION STUDY Evaluate relationship between variables to

determine if there is any correlation “Positive”, “Negative”, “Null”

Often miscategorize “association” with “cause” Positive correlation does not mean “A” causes “B”

Example: Obese persons drink soda, therefore soda causes obesity

Rely on memory recall or data documentation, which is often incomplete or unreliable

Used in surveys or retrospective chart reviews which may lead for a more controlled study

EPIDEMIOLOGICAL / CROSS-SECTIONAL SURVEYS

Collects data at a single point in time to evaluate a policy or public health issue Example: A case of hepatitis reported at

“Restaurant Q”, so all persons who ate at that restaurant are questioned to discover if they have symptoms consistent with hepatitis

Limited by event memory, difficulty in contacting subjects

EPIDEMIOLOGICAL / LONGITUDINAL OR CASE CONTROL STUDIES

Follows subjects over a period of time, serially asking research questions for study duration Example: Framingham Heart Study has

followed residents of Framingham, MA since 1948 looking for heart disease patterns

In case control studies, subjects with a condition are compared with those who do not Observational, not interventional

therefore classified as “epidemiological”

CASE REPORTS

Case Study / Case Report Detailed info about a single subject or a

small group of subjects

Examples: 1st time for new surgical technique, “weird” presentations of diseases

Case Series Report on a series of subjects with one

specific unifying factor but no control group

Examples: Outcomes of pelvic trauma patients with MAST trouser application

LITERATURE REVIEW

Exhaustive search of all relevant evidence-based literature related to a topic using multiple databases Cochrane reviews (meta-analysis)

PubMed (NIH study repository)

CINAHL

Asking a question of Google or Wikipedia is NOT an exhaustive (nor reliable) literature review

Accredited WikipediaResources

RETROSPECTIVE CHART REVIEWS / HISTORICAL STUDIES

Systematized searches for fact(s), then using information to describe, analyze & interpret the past

Example: Bledsoe BE, Smith MG. Medical Helicopter Accidents

in the United States: A 10-Year Review.

Methods: Review all US air medical helicopter accidents from 1997-2002 from NTSB database

Results: Majority of air medical helicopter accidents resulted from pilot error or mechanical failure

*J Trauma. 2004;56:1325-1329

EXPERT OPINIONS

Examples: Systematic reviews

Narrative reviews

Pure opinion pieces

Position papers

Based on a literature review (including RCT meta-analysis) but considered low on the “Scientific Evidence” schematic due to not being an actual “study”

RATIONALE CONJECTURE

Lowest level of scientific validity but often accurate & important

Anecdotal reports often fit into this category, as do “interesting observations”

Often amusing due to “duh” factor

“DUH” EXAMPLE: Back MD. Knowing Your Mate Value: Sex-Specific Personality Effects on Accuracy of Expected Mate Choice*.

Males rated photos of women displaying “exploitability cues” indicating appropriateness for one-night stands vs “stability cues” appropriate for marriage or long-term dating

22 exploitability cues correlated with “sexual receptivity”, including: Lip licking/biting, appearing intoxicated, tight clothing,

obesity, dumb conversations, rubbing breasts / genitals

Conclusions: Males prefer drunk, slutty**, dumb women for 1 night stands

**Authors’ words, not mine!

*Psychol Sci. 2011 Aug;22(8):984-9.

PSEUDOSCIENCE: THINGS WE BELIEVE ARE TRUE THAT AREN’T

• Epinephrine saves lives in OOHCA

• No RCT has shown better outcomes with epinephrine vs placebo in OOHCA

• Amiodarone improves outcomes in pediatric OOHCA

• Only 1 RCT suggests risk > benefit of amiodarone (not statistically significant)

• ATLS & the “Golden Hour”

• Evidence suggests ATLS has less impact on outcomes than believed

• Zero studies have shown anything magical about the 1st 60 minutes managing trauma patients

ARTICLE COMPONENTS

Author

Title

Key Words

Abstract

Introduction / Background

Methodology

Results

Discussion

Conclusion

References / Citations

Acknowledgements

AUTHOR INFORMATION

Primary, secondary & associate authors Primary & secondary do majority of work

Associates edit, collect data, produce diagrams

If a multi-center, 1 author from each site listed

If >10 sites, authors listed in acknowledgements

Lists primary site / contact information

Dirty secret…many “authors” have never seen the research – they are department chairs, “famous” or “friends of famous”, all of which lend credence to research (& help get it published)

TITLE

“Headline”* clearly & briefly describing what paper examines

May be misleading to attract attention

Example:

“Cellphones linked to brain cancer” is sexier than …

“Environmental risk factors for cancers of the brain and nervous system: the use of ecological data to generate hypotheses.”

*Keith Wesley

KEY WORDS

Important “search terms”

Different query results from: “Prehospital”

“Paramedics”

“EMTs”

“EMS”

ABSTRACT

Short summary of study

Distills key study elements into a few easily understood sentences

After reviewing, the reader determines if article is “worth my time to read”

INTRODUCTION / BACKGROUND

Familiarize readers with subject & research field

Examines key question & reviews current relevant literature

Provides rationale for why research is important

Presents hypothesis attempting to answer the key question

METHODOLOGY

Evaluate research quality by showing how study conducted Study design & population description Data collection & interpretation methods

Provides details so other researchers can replicate & validate findings

Reader “assumes” articles published in a peer-reviewed journal are statistically valid “Valid” data still may be misinterpreted by

the authors or media

RESULTS

Objective description of data indicating statistical significance / insignificance using raw & calculated data A “p value” <0.05 indicates result not likely to

have happened by chance, and is “probably true”

Data interpretation reserved for discussion section

Results must be interpreted by reader keeping in mind validity of the hypothesis, study methodology & author bias

DISCUSSION Study results interpreted & evaluated against existing body of literature

Attempts to connect results to bigger picture & show how results applied clinically Statistically significant findings may not have much clinical application

Author’s opinion about results is valuable but taken with a grain of salt Any bias or unexpected results reviewed & explained

No one starts a study expecting to “fail”

Example: If a author states “trend towards significance”, it means that the study failed to show a positive outcome, so the author “stretches the truth” to imply that data are really significant, but the study itself was flawed

Even well-designed / conducted studies limited by imperfect nature of EMS research

CONCLUSIONS

Summarizes info with recommendations based upon data analysis

Recommendations range from changing clinical practice to more research

Reader must weigh all parts of the study & determine if methodologically sound , clinically reproducible & valid enough to change or continue current practice

Race, gender, geography, economic status & “sensitive” factors should not be ignored due to political correctness Example: Research links asthma incidence & severity

with race/ socio-economic status; therefore providers should expect patients with more severe respiratory distress in urban, black & Hispanic areas vs rural Caucasian areas

REFERENCES / CITATIONS

Each cited author & paper plus quotes, ideas & content in a standard format

Source of further info on a topic Decisions rarely made after one study

References provide further info on that subject in which you are interested

WHAT DOES THAT CITATION MEAN?

Gottesman BE, Gutman A, et al. Radiation exposure in emergency physicians working in an urban ED: a prospective cohort study. Am J Emerg Med. 2010 Nov;28(9):1037-40. Epub 2010 Mar 9. PMID: 20825934

Authors / Researchers “And Others”; usually part of the research team

Title of study, describes the hypothesis & results

Journal Name; does not indicate if peer-reviewed

Print publication date: year, month; journal year(issue):pages

Electronic publication date: year, month, date

PubMed ID#

ACKNOWLEDGEMENTS & DISCLOSURES

Research expensive & often sponsored by a university or company

Authors must disclose financial or other support (i.e. airfare to a conference) to explain potential conflicts of interest

“Disclosures” indicate bias that may affect study’s outcome If design adheres to scientific method, little chance for bias to affect outcomes

Example: If a study determines a drug to be ineffective / harmful, & if that study were sponsored by a competitor of the drug then readers should know that there is not only bias present, but potentially a data misinterpretation

NEMSRA RECOMMENDATIONS TO IMPROVE FUTURE EMS RESEARCH

Structured training programs i.e. Fellowships

Collaborative Centers of Excellence combining investigators, resources & public policy i.e. The Resuscitation Outcomes Consortium (The “ROC”) i.e. Recognizing that prehospital care is an actual “academic” branch of medicine

Better evidence before implementing procedures, devices, or drugs i.e. More RCTs & evidence-based practice

Standardized data collection using uniform prehospital data elements i.e. Utstein criteria

The FDA & Office for Human Research Protections should re-evaluate exception from informed consent i.e. HIPPA and human subject testing exemptions

Investment in infrastructure to overcome obstacles impeding EMS research i.e. IT, funding, data-sharing

REFERENCES Pierce, L. L. Twelve steps for success in the nursing research journey. The Journal of Continuing Education in Nursing, 2009. 40, 154-162

My Athletic Life (Scientific Research 101 series). 2012. http://www.eecs.harvard.edu/~michaelm/postscripts

Bledsoe B. An Introduction to EMS Research (presentatiaon). 2009

www.researchagenda.org (National EMS Research Agenda)

www.naemsp.org (National Association of EMS Physicians)

www.acep.org (American College of Emergency Physicians)

www.rescuehouse.com (Rescue House Fire Resources)

www.nhtsa.gov (National Highway Transportation Safety Agency)

www.signal51group.com (Shreveport Fire Department Signal 51)

www.ci.shreveport.la.us

Edenburn D. Medical research made easy (presentation). 2013

Fox J, Day S, Reynolds L, et al. Post-clinical trial survey of EMS providers: Research experience and attitudes. Air Med J. 2010;29(1):34–36

Schmidt TA, Nelson M, Daya M, et al. EMS providers’ attitudes and experiences regarding enrolling patients in clinical research trials. PEC. 2009;13(2):160–165

US Department of Health & Human Services. Code of Federal Regulations 45 CFR46. 2010

US FDA Regulations Relating to Good Clinical Practice and Clinical Trials. 2011

Ryan KJ, Brady JV, Cooke MD, et al. Belmont Report: Ethical principles and guidelines for the protection of human subjects of research. In Office of Human Subjects Research. 1979

Literature / image searches: Wikipedia, Google, Bing, www.pubmed.com

SUMMARYwww.prehospitalmd@gmail.com / www.TEAEMS.com

EMS must be driven by science rather than conjecture & pseudoscience

Anyone can undertake a research project – just ask a question & try to scientifically find out the answer

EMS providers should analyze research & drawing appropriate conclusions

By understanding evidence-based medicine, providers play an active role in the advancement of patient care EMS must be driven by science