Assessment of methodological quality and outcomes of clinical trials

Christian Gluud

Copenhagen Trial Unit Centre for Clinical Intervention Research

RigshospitaletCopenhagen University Hospital

Evidence-basedclinical practice

• The patient’s values (concerns, expectations, preferences)

• The best clinical research evidence

Levels of evidence

Important aspects of clinical research

- all levels of the hierachy

• Systematic errors (domains) • Systematic errors (design) • Random errors (play of chance)

Levels of evidence

• Risk of systematic errors (domain)• Risk of systematic errors (design)• Risk of random errors

Important aspects of randomised clinical trials

and meta-analyses

• Systematic errors (domains)

• Systematic errors (design)

• Random errors (play of chance)

Domains associated with bias risk

• Generation of the allocation sequence• Allocation concealment• Blinding• Incomplete outcome data (intention-to-treat)• Outcome reporting bias• Industry bias• Other components associated with bias

Meta-analysis of several trials

Low risk of bias

High risk of bias

Overall

Ratio of odds ratios (ROR)Ratio of odds ratios (ROR)

Odds ratio of trials with unclear or inadequateOdds ratio of trials with unclear or inadequate

component (high risk of bias)component (high risk of bias)

divided bydivided by

odds ratio of trials with adequate odds ratio of trials with adequate

component (low risk of bias)component (low risk of bias)

Control of selection biasGeneration of the allocation sequence

• Low risk of bias Computer system, table of random numbers, or similar

• High risk of bias Not described or quasi-randomised (excluded)

BRANDO Sequence generation

• 112 meta-analyses with• 944 randomised clinical trials • 696 (73.7%) with high risk of bias• ROR 0.89 (95% CI 0.82 to 0.96)

Control of selection biasAllocation concealment

• Low risk of bias Central independent unit, sealed envelopes (only if perfect placebo)

• High risk of bias Not described or open table of random numbers

BRANDO Allocation concealment

• 146 meta-analyses with• 1292 randomised clinical trials • 916 (70.9%) with high risk of bias• ROR 0.93 (95% CI 0.87 to 0.99)

Control of detection bias(reporting bias and observer bias)

Blinding

• Low risk of bias Identical placebo or comparator

• High risk of bias Not described or not blinded

BRANDO Blinding

• 104 meta-analyses with• 1057 randomised clinical trials • 467 (44.2%) with high risk of bias• ROR 0.87 (95% CI 0.79 to 0.96)

Industry sponsorship and research outcome

• Lundh et al, The Cochrane Library 2012

• Cross-sectional studies, cohort studies, systematic reviews and meta-analyses that quantitatively compared primary research studies of drugs or medical devices sponsored by industry with studies with other sources of sponsorship

Favourable beneficial results

Favourable harms results

Favourable conclusions

Important aspects of randomised clinical trials

and meta-analyses

• Systematic errors (domains)

• Systematic errors (design)

• Random errors (play of chance)

Design errors affecting external validity of randomised trials include

‘wrong’: • Centres• Participants• Experimental intervention• Control intervention• Goal - explanatory or pragmatic• Trial structure - parallel group, crossover, etc.• Objective - superiority, equivalence, non-inferiority• Outcome• Unit of analysis

Important aspects of randomised clinical trials

and meta-analyses

• Systematic errors (domains)

• Systematic errors (design)

• Random errors (play of chance)

Mean number of patients per intervention arm (SEM) in 383 randomised trials

published in Gastroenterology from 1964-2000 (Kjærgard et al. 2002)

TA(C)E for hepatocellular carcinoma – trial sequential

analysis

The pernicious yin-yang interplay between random errors and systematic error

PUBLICATION BIASPUBLICATION BIAS

Number of patients with serum HBV DNA at end of treatment

Trial sequential analysis of serum HBV DNA

DARIS 4620 pts. Pc 54%, RRR 20%, alpha=5%, beta=20%, diversity=85%

Trial sequential analysis of serum HBV DNA at end of treatment

Number of pts without seroconversion from HBeAg to anti-HBe

DARIS 891 pts. Pc 86%, RRR 20%, alpha=5%, beta=20%, diversity 79%

Trial sequential analysis of seroconversion from HBeAg to anti-HBe

Trial variable Global Chinese

Trial registration Often insufficient Often insufficient

Sample size estimation 50% Less than 25%

Bias risks 95% More than 95%

Comparators Often ill chosen Often ill chosen

Outcomes Putative sourrogates Putative sourrogates

Results Mostly unreliable Mostly unreliable

Depesonolised data in register

Always insufficient Always insufficient

Experimental intervention- systematic errors (domains)- systematic errors (design)- PUBLICATION bias

Control intervention

Randomised clinical trials – most often false due to the people behind!

“…why most research findings are false!” JP Ioannidis

THANK YOU !THANK YOU !

The Cochrane

CollaborationAn international network of professionals,

preparing , maintaining, and disseminating systematic reviews of the effects of health care

Archie Cochrane (1979)

”It is surely a great criticism of our profession that we have not organised a critical summary, by speciality and subspeciality, adapted periodically, of all relevant randomised trials”

The Cochrane Collaboration

• www.cochrane.org• 670 000 randomised trials• Now 5000 systematic reviews• 500 new reviews per year• 500 updated reviews per year• JIF 6.2

Answer to the question:

When is it acceptable to make therapeutic decisions based on patient series, patient-control studies, and cohort studies?

Answer to the question:

When is it acceptable to make therapeutic decisions based on patient series, patient-control studies, and cohort studies?

Only if you have exophtalmus producing intervention effects!

Answer to the question:

When is it acceptable to make therapeutic decisions based on patient

series, patient-control studies, and cohort studies?

As we do not know the intervention effect a priory – and medical research is a

forward moving process – one should always test interventions in randomised clinical trials!

Thomas C Chalmers

• More than 500 publications• More than 25 000 citations• H index above 77

• Always randomize the first patient! The New England Journal of Medicine

1977;296(2):107

Rare diseases and randomised trials

• With 7,000,000,000 (7 billion!) individuals in the world the disease has to be extremily rare – say less than 1/10,000,000 – before it becomes difficult to conduct randomised clinical trials!

• International patient registries• Infrastructures like ECRIN

James Lind 1747

James Lind 1716-1794

James Lind allocated

Two oranges and lemons

Two cider

Two vinegar

Two elixir vitriol

Two spices and garlic

Two sea water

75 498 979 1647 31188823

16155

26322

45020

67522

99373

121584

131223

0

20000

40000

60000

80000

100000

120000

140000

1900 to 1946

1947 to 1951

1952 to 1956

1957 to 1961

1962 to 1966

1967 to 1971

1972 to 1976

1977 to 1981

1982 to 1986

1987 to 1991

1992 to 1996

1997 to 2001

2002 to 2007

Year period

Periodic growth of publications on randomised and controlled clinical trials

1127

1014945

881

713666 632

594 587 559

438 430 426355 349

254 250199 188 175

0

200

400

600

800

1000

1200

Number of publications on randomised and controlled clinical trials published per million inhabitants

James Lind observed

The two on oranges and lemons

regained health in a few days

and could care for the remainder,

that stayed ill

James Lind 1753

James Lind 1747

Yearly ‘International Clinical Trials’ Day’ !!!!!Yearly ‘International Clinical Trials’ Day’ !!!!!

George Löhner and…... 1835

George Löhner and his team of truth loving men

• Clear protocol, published before launch

• Large number of participants (n = 50)

• Perfect randomisation

• Placebo controlled (melted ice)

• Blinded for all parties

• Account for drop outs

• Statistical comparison

Downgrade the evidence in case of risks of errors

I More evidence needed!!!!!!!!!!!!!IIa Systematic review of randomised clinical trials with risk of systematic errors (bias) or of random errors (play of chance) IIb Single randomised clinical trial with risk of systematic errors (bias) or of random errors (play of chance) III Cohort studyIV Case-control studyV Case series

Upgrade the evidence in case of ‘exophthalmus’ producing effects

I Case-control studiesII Case series

Ex: - parachute for plane passengers - insulin for diabetic coma- blood transfusion for severe haemorrhage- defibrillation for ventricular fibrillation- ether to induce anaesthesia

Povl Heiberg 1897

Heiberg knew the solutions

- large numbers - large numbers !!

- randomisation - randomisation !!

- blinding - blinding !!

- independent research - independent research !!

The hierarchy of evidenceIa Systematic review of randomised clinical trials with low risk of systematic errors (bias) and of random errors (play of chance) Ib Single randomised clinical trial with low risk of systematic errors (bias) and of random errors (play of chance) II Cohort studyIII Case-control studyIV Case series

Low risk of systematic errors

High risk of systematic errors

Low risk of random errors

High risk of random errors

Random errors in small trials• False positive results

(type I error)

• False negative results

(type II error)

Components associated with bias risk

• Generation of the allocation sequence• Allocation concealment• Blinding• Incomplete outcome data (intention-to-treat)• Outcome measure reporting bias• Other components associated with bias (vested

interest bias)

TA(C)E for hepatocellular carcinoma – systematic review

Important aspects of randomised clinical trials

and meta-analyses

• Systematic errors (bias) • Random errors (play of chance)• Design errors

The hierarchy of evidenceIa Systematic review of randomised clinical trials with low risk of systematic errors and low risk of random errors Ib Single randomised clinical trial with low risk of systematic errors and low risk of random errors !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!II Cohort studyIII Case-control studyIV Case series

Dangers in non-randomised studies

• Confounding by indication and unmeasured confounding • Biological mechanisms - Limited time of diseases - Cyclical progression of diseases - When do we see patients?

• Psychological mechanism - We see what we want to see (BIAS)! - We believe what we want to believe (astrology)!

Deeks et al (2003) and the International Stroke Trial

• 19,435 patients with ischaemic stroke

• 467 clinical sites

• Randomised to aspirin versus placebo

• Dead or dependent at 6 months

• OR 0.95; 95% CI 0.89 to 1.01

Deeks et al (2003) and the International Stroke Trial

Deeks et al. resample 100 pts/group making

- Small randomised trials (n=14,000) from

randomised groups within centres

- Small controlled cohort studies (n=14,000)

from two centres

Deeks et al. 2003

Randomised clinical trials

Deeks et al. 2003

Controlled cohort studies

Deeks et al. 2003

Controlled cohort studies

Logistic regression

Deeks et al. 2003

Controlled cohort studies

Propensityscores

Meta-analysis of several trials

Low risk of bias

High risk of bias

Overall

Ratio of odds ratios (ROR)Ratio of odds ratios (ROR)

Odds ratio of trials with unclear or inadequateOdds ratio of trials with unclear or inadequate

component (high risk of bias)component (high risk of bias)

divided bydivided by

odds ratio of trials with adequate odds ratio of trials with adequate

component (low risk of bias)component (low risk of bias)

Control of selection biasGeneration of the allocation sequence

• Low risk of bias Computer system, table of random numbers, or similar

• High risk of bias Not described or quasi-randomised (excluded)

BRANDO Sequence generation

• 112 meta-analyses with• 944 randomised clinical trials • 696 (73.7%) with high risk of bias• ROR 0.89 (95% CI 0.82 to 0.96)

Control of selection biasAllocation concealment

• Low risk of bias Central independent unit, sealed envelopes (only if perfect placebo)

• High risk of bias Not described or open table of random numbers

BRANDO Allocation concealment

• 146 meta-analyses with• 1292 randomised clinical trials • 916 (70.9%) with high risk of bias• ROR 0.93 (95% CI 0.87 to 0.99)

Control of detection bias(reporting bias and observer bias)

Blinding

• Low risk of bias Identical placebo or comparator

• High risk of bias Not described or not blinded

BRANDO Blinding

• 104 meta-analyses with• 1057 randomised clinical trials • 467 (44.2%) with high risk of bias• ROR 0.87 (95% CI 0.79 to 0.96)