The material was supported by an educational grant from Ferring How to critically appraise a study...

The material was supported by an educational grant from Ferring

How to critically appraise a study

Nikolaos P. Polyzos M.D. PhD

Learning objectives

• At the end of this presentation you should appreciate that you should not accept the conclusions of any paper without question.

• This presentation will provide guidance of how you should question the conclusions of papers.

A research finding is less likely to be true when:

• the studies conducted in a field are smaller

• effect sizes are smaller

• there is greater flexibility in designs, definitions, outcomes, and analytical modes

• there is greater financial and other interest and prejudice

Ioannidis JP, Plos Med 2005

Essay: Why Most Published Research Findings Are False - John P.A. Ioannidis

• Why you should question the results in all papers• How you should question the results

Top cited articles are not ALWAYS telling

the truth

Why you should question the results in all papers

• Initial findings…might prove wrong in the future

• Highly cited studies (>1000 citations) with efficacy claims

– 16% were contradicted by subsequent research

– 16% were found to have initially stronger effects

– 44% were replicated also with a larger sample size in subsequent research compared with the original highly cited study)

– 24% had remained largely unchallenged

Top cited articles are not always telling the truth

Ioannidis JP, JAMA 2005

Data fabrication can occur even in highly esteemed journals

Why you should question the resultsin all papers

Sudbø case (Lancet)

Among 908 patients included in the study 250 had the same birth date

Data fabrication can occur even in highly estimated journals

Low quality randomized trials may show inflated treatment effect


• Cochrane Pregnancy and Childbirth Database – 250 controlled trials from 33 meta-analyses

• Greater apparent treatment effect in studies of poorer quality– Randomisation issues

• Odds Ratios exaggerated up to 41%

– Inadequate blinding • Odds Ratios exaggerated by 17%

Low quality randomized trials may show spuriously inflated treatment effect

Schulz. JAMA 1995; 273 (5): 408–412

META-ANALYSIS PLOT FOR PRETERM BIRTH <37 WEEKS OF GESTATION

Study or subgroup Treatment No treatment weight Odds ratioEvents/toal Events/total (%) (M-H, fixed)

(95% CI)Low quality trailsSubtotal (95% CI) 114/996 147/725 35.2 0.52 (0.38 to 0.72)

Test for heterogeneity: χ2 = 4.95, df = 5, P = 0.42, I2 = 0%

Test for overall effect: z = 4.01, P<0.001

High quality trialsSubtotal (95% CI) 250/2303 219/2290 64.8 1.15 (0.95 to 1.40)

Test for heterogeneity: χ2 = 4.02, df = 4, P = 0.40, I2 = 1%

Test for overall effect: z = 1.45, P = 0.15

Total (95% CI) 364/3299 366/3015 100 0.93 (0.79 to 1.10)Test for heterogeneity: χ2 = 25.94, df = 10, P = 0.004, I2 = 61%

Test for overall effect: z = 0.86, P = 0.39

M-H=Mantel-Haenszel fixed effects model0.1 0.2 0.5 1 2 5 10

Favours treatment Favours no treatment

Odds ratio(M-H, fixed)

(95% CI)

Low quality trials showed treatment effect that was not confirmed in high quality studies

Polyzos NP et al., BMJ 2010

Trials with negative results are not published or get published later


Among studies presented in major scientific meetings, trials with positive results are more likely to get published compared to negative studies

0 20 40 60 80 100

1.0

0.8

0.6

0.4

0.2

0.0

Time to publication (months)

Pro

po

rtio

n o

f a

bst

rac

ts p

ub

lis

hed

in

fu

ll-t

ext

Outcome in favour of the experimental arm

Not-positive

Positive

Not-positive-censored

Positive-censored

Trials with negative results are not published or get published later

Polyzos NP et al., Hum Reprod 2011


Meta-analyses often rely on few or inconsistent evidence

• 61 systematic reviews published during July 2012 in Cochrane– 15% of the reviews included 1or 0 trials

– Half included fewer than 1,000 patient randomized patients

– 31 were updated reviews

– 11 of these 31 updated reviews included the same number of trials and participants as the previous review they sought to bring up to date.

Meta-analyses and systematic reviews may often rely on few or inconsistent evidence

Humaidan & Polyzos Nat Med 2012

Industry supported research may demonstrate greater treatment effects



Lexchin J et al., BMJ. 2003.OR 4.05; 95% CI 2.98-5.51

Research funded by drug companies was 4x more likely to have outcomes that favour the sponsor's product than research funded by other sources

Study (first author) Odds ratio

0.1 0.2 0.5 1 2 5 10 100 1000 10000

Azimi12

Cho14

Clifford15

Davidson16

Dieppe18

Djulbegovic19

Djulbegovic20

Friedberg23*

Friedberg23┼

Kamal-Bahl26╪

Kamal-Bahl26§

Koep30

Mandelkern32

Sacristan36¶

Sacristan36**

Thomas38

Vandenbroucke39

Yaphe41

82% of industry sponsored cost-effectiveness analyses show that drugs are cost effective


Valachis et al., J Clin Oncol 2012

Cost-effectiveness analyses with industry involvement were associated with lower baseline assumptions of the sensitivity of the Pap test.

The “Straw-man” comparator

100

90

80

70

60

50

40

30

20

10

00 10 20 30 40 50 60 70 80 90 100

Author affiliation or funding or conflict of interest |with the manufacturer

Estimates for CEA’s without authors affiliated, funding or conflict of interest with the manufacturerEsitmates for CEA’s with at least one author affiliated with the manufacturer or with funding or with conflict of interest with the manufacturer

1 - Specificity

Sen

siti

vity


Polyzos NP et al., CMAJ 2011


Industry sponsored trials and even government sponsored trials have been published late in

the past

• Time to publication – Industry (24months) vs. non-industry (20 months) after study

completion p<0.001Ross JS et al., JAMA 2013

• Publication rates within 2 years – industry (40%) vs. non-government/non-industry (56%) – RR = 0.73, 95% CI 0.61–0.87; p<0.001– industry (40%) vs. government trials (47%); p = 0.22

Ross JS et al., Plos Med 2009

Industry sponsored trials may get published later than non sponsored trials

• NIH trials NOT PUBLISHED after completion – 30 months - half unpublished – 51 months - 1/3 unpublished

Even government sponsored trials fail to get published soon

80

60

40

20

00 20 40 60 80 100

Time from study completion (months)

Per

cent

age

of s

tudi

es p

ublis

hed

No at risk

635 635 635 635 493 330 220 153 95 54 44

No of unpublished studiesTrial completed before 2007

269 264 259 235 221 197 175

Trial completed in 2007-8

366 356 324 282 244 215 176

80

60

40

20

00 5 10 15 20 25 30

Time from study completion (months)

Per

cent

age

of s

tudi

es p

ublis

hed

P<.001

Trial completed before 2007

Trial completed in 2007-8

Ross JS et al., BMJ 2011

• Why you should question the results in all papers• How you should question the results

How you should question the results

Do not upfront reject a study based on its status as clinical trial governance and reporting have

improved over time

Less likely to even want to read the full article !

Industry connection to a publication reduces its value to physicians

Industry sponsorship:• negatively influences physicians' perception of methodological quality• reduces their willingness to believe and act on trial findings, independently of the trial's quality

1.25

1.00

0.75

0.50

Industry vs. None Industry vs. NIH

Funding

Od

ds

Rat

io

0.68 (95% CI, 0.49-0.94)P = .02

0.52 (95% CI, 0.37-0.71)P<.001

Physician Willingness to Prescribe Drug

Do not upfront reject a study based on its status

Kesselheim NEJM 2012

• The Good Clinical Practice Directive (Directive 2005/28/EC of 8 April 2005 of the European Parliament and of the Council)

• EudraCT vs. 9 (European trials registry)• Results must be provided for all interventional clinical

trials which commenced in the European Union from 01 May 2004 onwards

• The reporting and public disclosure includes:– Trials which have terminated early– Trials with positive as well as negative trial results– Trials of products with OR without a marketing authorisation within

the community

https://eudract.ema.europa.eu/index.html

Why you should not upfront reject a study based on its status now



• For Pharma to introduce a new drug the drug must pass Phase II studies to establish dosing and safety, followed by Phase III studies to confirm efficacy and further demonstrate safety which are externally monitored and must follow high methodological studies

• Phase II success rates for new development projects have fallen from 28% (2006–2007) to 18% (2008–2009)

• The combined success rate at Phase III and submission has fallen to ~50% from 2007 to 2010

Has the situation been improved over time?

Arrowsmith. Nature Reviews Drug Discovery 2011

Arrowsmith. Nature Reviews Drug Discovery 2011

Do not upfront reject a study based on its status

How you should question the results

Use a standard critical appraisal approach when evaluating a scientific article

Critically evaluate the purpose of the study• What is the rationale for performing the study• Is/are the research question(s) clearly defined and if not, should they

be?

Framework for How to Read and Critique a Research StudyAmerican Nurses Association http://www.nursingworld.org

Use a standard critical approach when evaluating a scientific article

Critically evaluate the design of the study• Is the design appropriate for the study?• Does the sample fit with the research design and is the size sufficient? • How were data collected? • Is the analytical approach consistent with the study questions and

research design?



Examine whether the study has been registered in a public registry • Has the trial been registered prior starting the trial?• Are the primary endpoints the same with the registered version of the

trial?


Always review the existing literature• Is the literature review relevant to the study, comprehensive, and

include recent research? • Does the literature review support the need for the study? • Are the findings consistent with existing literature?



Critically evaluate the results and conclusions• Are the results presented clearly in the text, tables and figures? • Are the statistics clearly explained? • Are the results explained in relationship to the theoretical framework,

research questions?• Are the limitations presented and their implications discussed? • Are there recommendations for clinical practice



• No trial is without limitations, results may occur by chance

• There is even a risk results may be misleading • The more robust the study the more likely it is to be true• Always review all the literature and look for consistency

and inconsistency in what is reported• Do not upfront accept or reject the results of a study just

based on the journal published, research group or industry involvement

Conclusions

Thank you

The material was supported by an educational grant from Ferring How to critically appraise a study...

Documents

Transcript of The material was supported by an educational grant from Ferring How to critically appraise a study...