8/6/2019 How to Evaluate an Article
1/38
Dr. Noha A. DashashMBBS, DPHC, ABFM, SBFMConsultant Family Physician,Director of PHC, Jeddah GovernorateMember of the National EBM CommitteeSupervisor, Jeddah EBM Working Group
Dr. Noha A. DashashMBBS, DPHC, ABFM, SBFMConsultant Family Physician,Director of PHC, Jeddah GovernorateMember of the National EBM CommitteeSupervisor, Jeddah EBM Working Group
8/6/2019 How to Evaluate an Article
2/38
29 May 2011 EBM Jeddah Working Group 2
In The Name Of AllahThe Most Merciful
The Most Compassionate
8/6/2019 How to Evaluate an Article
3/38
29 May 2011 EBM Jeddah Working Group 3
Components of an articleComponents of an article
Title
Abstract Introduction-background
Methods (Materials & Methods)
Results Discussion- Conclusion
8/6/2019 How to Evaluate an Article
4/38
29 May 2011 EBM Jeddah Working Group 4
IMRADIMRAD
Introduction
why
Methodology how
Results and
What they found Discussion
What they THINK the results mean
8/6/2019 How to Evaluate an Article
5/38
29 May 2011 EBM Jeddah Working Group 5
The science of trashing aThe science of trashing a
paperpaper Does it answer your question?
Title
Abstract
Introduction
AIM, Objectives
Study Design: Is it suitable for answering your question?
8/6/2019 How to Evaluate an Article
6/38
29 May 2011 EBM Jeddah Working Group 6
The science of trashing aThe science of trashing a
paperpaper
METHODS
If methodology is not sound
trash the paper before reading the
results even if statistically
significant
8/6/2019 How to Evaluate an Article
7/38
29 May 2011 EBM Jeddah Working Group 7
Critical AppraisalCritical Appraisal
critique
1. A critical review or commentary,
especially one dealing with worksof art or literature.
2. A critical discussion of a
specified topic.3. The art of criticism.
The Free dictionary, by Farlex
8/6/2019 How to Evaluate an Article
8/38
29 May 2011 EBM Jeddah Working Group 8
Critical AppraisalCritical Appraisal
critique
review, oranalyze
a serious examination andjudgment of something;
"constructive criticism is alwaysappreciated"
The Free dictionary, by Farlex
8/6/2019 How to Evaluate an Article
9/38
29 May 2011 EBM Jeddah Working Group 9
Critical AppraisalCritical Appraisal
Appraisal: the classification of
someone or something with
respect to its worth
8/6/2019 How to Evaluate an Article
10/38
29 May 2011 EBM Jeddah Working Group 10
Critical AppraisalCritical Appraisal
Critical appraisal: an appraisal
based on careful analytical
evaluation
8/6/2019 How to Evaluate an Article
11/38
29 May 2011 EBM Jeddah Working Group 11
(critically appraise) evaluation for
validity and usefulness
8/6/2019 How to Evaluate an Article
12/38
29 May 2011 EBM Jeddah Working Group 12
8/6/2019 How to Evaluate an Article
13/38
29 May 2011 EBM Jeddah Working Group 13
Measures of StudyMeasures of StudyReliabilityReliability
B ias, I nternal validity,
C onfounding,
E xternal validity,
P ower
8/6/2019 How to Evaluate an Article
14/38
29 May 2011 EBM Jeddah Working Group 14
BiasBias
A systematic tendency to produce anoutcome that differs from the
underlying truth
factorarisingfrom Design
Conductofa study
skewsthe data in one particulardirection,eitherawayfromortowardsthetrue
population valuethatis beingestimatedwiththestudy
8/6/2019 How to Evaluate an Article
15/38
29 May 2011 EBM Jeddah Working Group 15
BiasBias
Selection bias
example, we select sicker patients to
receive active treatment and fitter
patients to receive inactive treatment
How can it be overcome? Randomization CONCEALMENT
8/6/2019 How to Evaluate an Article
16/38
29 May 2011 EBM Jeddah Working Group 16
BiasBias
Observer bias
example, we know that the patient
had active treatment so wesubconsciously encourage her to rateher quality of life as higher than itreally is.
How can it be overcome? Blinding
8/6/2019 How to Evaluate an Article
17/38
29 May 2011 EBM Jeddah Working Group 17
BiasBias
Participant bias
example, in a study of aspirin versus
no treatment, people allocated to notreatment take aspirin anyway
Contamination
8/6/2019 How to Evaluate an Article
18/38
29 May 2011 EBM Jeddah Working Group 18
BiasBias
Withdrawal bias ordrop out bias
when we lose people to follow up,
those that remain for analysis at theend of the study may not be
representative of the group originally
included at the start of the study.
8/6/2019 How to Evaluate an Article
19/38
29 May 2011 EBM Jeddah Working Group 19
BiasBias
Recall bias
example, mothers of children withleukemia may remember living nearhigh voltage power cables becausethey fear a link between power linesand cancer, while mothers of childrenwithout leukemia are likely to forget
whether they lived near a power line,because they regard it as a trivial fact.
8/6/2019 How to Evaluate an Article
20/38
29 May 2011 EBM Jeddah Working Group 20
BiasBias
Instrument or measurement bias
E.g. blood pressure measurement
8/6/2019 How to Evaluate an Article
21/38
29 May 2011 EBM Jeddah Working Group 21
BiasBias Publication bias (in Systematic reviews/MA)
results from researchers and journalsbeing biased towards publishing onlypositive results
If negative or equivocal results remainhidden, then clinicians can get amisleadingly excessive view of howeffective a treatment really is (or of how
harmful an exposure might be).
authors of good systematic reviewsmake a point of tracking downunpublished research.
8/6/2019 How to Evaluate an Article
22/38
29 May 2011 EBM Jeddah Working Group 22
WorkWork--up Biasup Bias (Verification Bias)(Verification Bias)
In Studies of Diagnostic Tests:
Subjects with a negative test are sometimes not
referred to the 'gold standard' assessment.
In some cases, the investigators may refer a small
random sample of subjects with a negative test to
the 'gold standard' assessment in order to
determine if they really do not have the disease.
'gold standard' results are not available on anentire population,
work-up bias,
equations for sensitivity and specificity give biased
results.
8/6/2019 How to Evaluate an Article
23/38
29 May 2011 EBM Jeddah Working Group 23
Internal validity ofInternal validity ofmeasurementmeasurement
how well a study was run (researchdesign, operational definitions used, howvariables were measured, what
was/wasn't measured
Wasitreallythetreatmentthatcausedthe difference between thesubjectsin
thecontrol andexperimental groups?"In descriptive studies (correlational, etc.)
internalvalidityrefersonlytotheaccuracy/qualityofthestudy
8/6/2019 How to Evaluate an Article
24/38
29 May 2011 EBM Jeddah Working Group 24
results of study are due to the unusual typeof people who were in the study.
Or, it might only work because of the unusualplace you did the study in
(e.g. you did your educational study in a collegetown with lots of high-achieving educationally-oriented kids).
Or, study done in a peculiar time. For instance, smoking cessation study done the
week after the General Surgeon issues the well-publicized results of the latest smoking and cancerstudies,
you might get different results than if you had doneit the week before.
8/6/2019 How to Evaluate an Article
25/38
29 May 2011 EBM Jeddah Working Group 25
Internal validity ofInternal validity ofmeasurementmeasurement
1. How good (VALIDITY of) ourassessment tools are for measuring
the things we want to measure in thestudy (outcomes)
Quality of life: asking people how are
you on a scale of 1 to 10?
or
Using a validated Questionnaire(e.g.QALY)
8/6/2019 How to Evaluate an Article
26/38
29 May 2011 EBM Jeddah Working Group 26
Internal validity ofInternal validity ofmeasurementmeasurement Cont.
Glucose control assessment
monthly urine dipstick test?
Or
glycosylated hemoglobin
8/6/2019 How to Evaluate an Article
27/38
29 May 2011 EBM Jeddah Working Group 27
Internal validity ofInternal validity ofmeasurementmeasurement Cont.
Smoking
Asking people how much do you smoke?
Or
asking people to keep their empty
cigarette packets for measuring tobacco
exposure.
Whichhasmoreinternalvalidity
8/6/2019 How to Evaluate an Article
28/38
29 May 2011 EBM Jeddah Working Group 28
Internal validity ofInternal validity ofmeasurementmeasurement Cont.
2. Proxy or intermediate endpoints For example a study may suggest that a treatment
reduces heart disease, when the only outcome
assessed was blood cholesterol.
The internal validity of cholesterol measurements for
measuring heart disease is poor
8/6/2019 How to Evaluate an Article
29/38
29 May 2011 EBM Jeddah Working Group 29
ConfoundingConfounding
is an exposure that the
researcherdid notmeasure that
independently affectstheoutcomes of the study andwhich
is associated with the exposure
8/6/2019 How to Evaluate an Article
30/38
29 May 2011 EBM Jeddah Working Group 30
ConfoundingConfounding
Example: study of the relation between lung
cancer and coffee drinking
association is found between the two.
cause coffee drinkers are more likely to
smoke
real risk factor for lung cancer
smoking. In this study, the left-out variable ofsmoking
confounds the relationship between coffee
and lung cancer
8/6/2019 How to Evaluate an Article
31/38
29 May 2011 EBM Jeddah Working Group 31
External validityExternal validity
External validity is related to
generalizing
8/6/2019 How to Evaluate an Article
32/38
29 May 2011 EBM Jeddah Working Group 32
Threats to External ValidityThreats to External Validity
8/6/2019 How to Evaluate an Article
33/38
29 May 2011 EBM Jeddah Working Group 33
Threats to External ValidityThreats to External Validity
if you did your smoking cessation study the
week after the Surgeon General issues the
well-publicized results of the latest smoking
and cancer studies
type of people who were in the study
8/6/2019 How to Evaluate an Article
34/38
29 May 2011 EBM Jeddah Working Group 34
PowerPower
the ability of the study to detect
an effect if in truth there is an
effect (ie were we to perform an ideal studyof the entire population). We can
increase the power of the study
by:
8/6/2019 How to Evaluate an Article
35/38
29 May 2011 EBM Jeddah Working Group 35
We can increase the power ofWe can increase the power ofthe study by:the study by:
(a) increasing the number of
outcome events we observe
by increasing the sample size
or duration of the study
8/6/2019 How to Evaluate an Article
36/38
29 May 2011 EBM Jeddah Working Group 36
We can increase the power ofWe can increase the power ofthe study by:the study by:
(b) improving the precision of our
measurements
by increasing the sample size
or duration of the study
8/6/2019 How to Evaluate an Article
37/38
29 May 2011 EBM Jeddah Working Group 37
In a NutshellIn a Nutshell
8/6/2019 How to Evaluate an Article
38/38
THANK YOUTHANK YOU