research synthesis seminar revised - vghtpefdc.weebly.com · Research Synthesis: How to conduct a...

Research Synthesis:How to conduct a systematic review and meta‐analysis for

evidence‐based knowledge and policy

Rintaro Mori MD PhD MSc FRCPCHCochrane Japan / National Center for Child Health and Development

[email protected]

SIDSAn example

SIDS(Sudden Infant Death Syndrome)

History of SIDS studies

Odds ratio.1 1 10

Hauck

Brooke

Oyen

Fleming2

Irgens

Jorch

Ponsonby

Hoffman

Wigfield

Mitchell

Engelberts

Dwyer

Bouvier-Colle

Fleming1

de Jonge

McGlashan

Lee

Nicholl

Beal2

Beal1

Senecal

Cameron

Tonkin

Froggatt

Carpenter 19651970

19861987

1988

1989

1990

1991

19921993199419951996

19972004

CumulativeMeta‐analysis

Incidences of SIDS and Years of national campaigns

0

0.5

1

1.5

2

2.5

per 1000 live births

Incidences of SIDS

UK

Japan

Australia

US

UK

AUS

US

JP


Odds ratio.1 1 10

Hauck

Brooke

Oyen

Fleming2

Irgens

Jorch

Ponsonby

Hoffman

Wigfield

Mitchell

Engelberts

Dwyer

Bouvier-Colle

Fleming1

de Jonge

McGlashan

Lee

Nicholl

Beal2

Beal1

Senecal

Cameron

Tonkin

Froggatt

Carpenter 19651970

19861987

1988

1989

1990

1991

19921993199419951996

19972004



Odds ratio.1 1 10

Hauck

Brooke

Oyen

Fleming2

Irgens

Jorch

Ponsonby

Hoffman

Wigfield

Mitchell

Engelberts

Dwyer

Bouvier-Colle

Fleming1

de Jonge

McGlashan

Lee

Nicholl

Beal2

Beal1

Senecal

Cameron

Tonkin

Froggatt

Carpenter 19651970

19861987

1988

1989

1990

1991

19921993199419951996

19972004


Randomised TrialAnother example

Randomised Controlled Trials

Control group

Intervention group Follow‐up

Follow‐up

Compare groups

Enrolment

Random assignment

Randomised Studies & Non‐randomised Studies

A systematic review of comparisons of effect sizes derived from randomised and non‐randomised studies、HTA 2000

○:RCT◆: Non‐RCT

Structure of Systematic Review


Defining Research Question

• PICO Framework• Population

• Intervention

• Comparator

• Outcome


Literature Searching

• Practical Session

• Based upon your PICO


Study Design

• Interventional Studies• Randomised Controlled Trials

• Quasi‐randomised Controlled Trials

• Observational Studies• Cohort Studies

• Case‐control Studies

• Cross‐sectional Studies

• Case‐series

• Ecological Studies

Quality of Studies

• Internal Validity• Chance

• Bias

• Confounders

• Measurement Errors

• External Validity• Population

• Intervention/Exposure

• Comparator

• Outcome


Meta‐analysis

• Practical Session

Searching literature

Searching Skills as a specialty

• Information Specialist• Librarian by background

• Additional training in health information

• Using databases

• Developing search strategies

Information Specialist

Trained MD

70% 50%

Finding all the relevant literature

Databases for Health

• Pubmed/Medline

• Embase

• Popline

• Cochrane Library

• Web of Knowledge

• LILACS

• CAB abstracts

• African Health Line

• PsycInfo

• IBSS

• SIGLE

Databases for health in resource limited settings• Development Databases

• Popline

• Source Databases

• Pubmed

• Cochrane Library

• TRIP Database

• Latin American Database (LILACS)

• CAB abstracts

Characteristics of major databases

• Pubmed(Medline)• 1966 onwards

• Including non clinical aspects

• Western (American)‐oriented

• Strong English‐language bias

• Over 4500 journals from 70 countries


• Popline• Reproductive and sexual health

• Fertility/family planning

• Population issues

• Including journal articles, reports, unpublished materials, books and conference papers

• Excellent coverage for resource limited settings


• Cochrane Library• Full text systematic reviews

• Abstracts of reviews of effectiveness

• Clinical trials

• Economic evaluations


• EMBASE• Similar to Medline

• European emphasis

• Strong on drug/pharmaceutical literature


• LILACS• Covers literature related to the health sciences published in countries of Pan‐American Region since 1982

• Over 15000 records from 670 medical journals

MeSH

• http://www.ncbi.nlm.nih.gov/sites/entrez

Newbon

１．Newborn

１．Newborn

２．Go

# Search History Results

1 INFANT, PREMATURE/ 30667

2 preterm$.tw. 26998

3 INFANT, NEWBORN/ 398397

4 (newborn$ or neonate$).tw. 130516

5 or/1‐4 453023

6 HYPERBILIRUBINEMIA/ 3297

7 HYPERBILIRUBINEMIA, NEONATAL/ 125

8 hyperbilirubin?emia$.ti. 2101

9 bilirubin?emia$.ti. 146

10 ((bilirubin$ or hyperbilirubin$) adj3 encephalopath$).tw. 262

11 exp JAUNDICE/ 9418

12 jaundice$.ti. 9291

13 KERNICTERUS/ 852

14 kernicterus$.ti. 341

15 or/6‐14 19480

16 PREVALENCE/ 107283

17 prevalen$.ti. 49745

18 INCIDENCE/ 111887

19 inciden$.ti. 58452

20 RISK/ 75807

21 relative risk$.ti. 666

22 ODDS RATIO/ 30175

23 (risk ratio or odds ratio).ti. 258

24 TIME FACTORS/ 758296

25 (duration or time period or length of time).ti. 17154

26 MORTALITY/ 27533

27 INFANT MORTALITY/ 21113

28 (mortality rate$ or death rate$).ti. 3086

29 outcome$.ti. 85491

30 global burden.tw. 547

31 impairment$.tw. 122136

32 sequelae.tw. 32164

33 disability.tw. 46579

34 or/16‐33 1387101

35 and/5,15,34 558

36 limit 35 to humans 529

Research Question

• Population: Newborn requiring resuscitation

• Intervention: Resuscitation with 21% Oxygen

• Comparator: Resuscitation with 100% Oxygen

• Outcome: Mortality, disability, etc

In newborn infants requiring neonatal resuscitation, is resuscitation with 21% oxygen(room air), compared with that with 100% oxygen, effective to reduce mortality and

morbidities in them?

Keywords

• Newborn

• Resuscitation

• Oxygen

• Air

• Randomized controlled trial

Synonym?

• Newborn(s), neonate(s), infant(s)

• Resuscitation, cardiac massage, life support, emergency treatment(s)

• Oxygen, O2, air, gas

• Randomiz(s)ed controlled trial(s)/study(ies), RCT

MeSH

• Sets of terms naming descriptors in a hierarchical structure that permits searching at various levels of specificity

• ~26,000 terms (updated annually)

• Over 160,000 entry terms

Common Strategy

Searching results for Population

Searching results for Intervention

(Searching results for Study Design)

Analysing studies

Study Design

• Interventional Studies• Randomised Controlled Trials

• Quasi‐randomised Controlled Trials

• Observational Studies• Cohort Studies

• Case‐control Studies

• Cross‐sectional Studies

• Case‐series

• Ecological Studies

Study Designs

Study Design? Analytic?

Analytic Studies

Intervention

Randomisedcontrolled studies

Non‐randomisedstudies

Observational

Cohort studies

Cross‐ sectional studies

Case‐ controlled studies

Descriptive Studies

Qualitative studies

Prevalence/ incidence surveys

Randomised Control Trial

• Comparison of interventions

• Based on statistical power of randomisation

• Expensive

• Open to bias

RCT

Control group

Intervention group Follow‐up

Follow‐up

Compare groups

Enrolment

Random assignment

Intervention/Comparator

OUTCOME

Cohort Study

• Comparison of groups, usually based on single difference

• Follow‐up for long periods

• Prospective

• Not randomised

• Difficult to monitor people over long periods

Cohort Study

Comparison group

Group of interest

Follow over time

Follow over time

Compare outcomes

EXPOSURE OUTCOME

Case‐Control Study

• Comparison of group of interest against matched control group based on single factor – such as intervention

• Useful when examining rare conditions

• Retrospective

• Only based on non‐cases

Case‐Control Study

Take histories

Take histories

Compare histories

Group of interest

Comparison group

Draw conclusion

EXPOSURE OUTCOME

Case‐Series

• Based on medical records

• Open to selection bias

• No comparison group

Case‐Series

Patients Records

Report

EXPOSURE OUTCOME

No Control Group!

Cross‐Sectional Study

• Examine cohort at single time period

• Useful in epidemiology and public health

Cross‐Sectional Study

Population surveyed at one point in time

Population

Condition of interest

EXPOSURE

OUTCOME

Quality of Studies

• Internal Validity• Chance

• Bias

• Confounders

• Measurement Errors

• External Validity• Population

• Intervention/Exposure

• Comparator

• Outcome

Study 1

• A “continuous support in labour by midwives” policy has been implemented in some areas in the UK. Other areas have not had such services yet. A total of 1500 pregnant women in labour all over the UK were enrolled in this study at 20 week gestation. The women were interviewed at enrolment, soon after birth, at 2 weeks, 6 weeks, 6 months and 13 months, as well as tracking case notes of mothers and babies. Exposure status was defined as with or without continuous one‐to‐one support by midwives. 500 women had such support in labour, and the rest did not have. The women did not have a choice. Main outcome measures include intervention rates, maternal satisfaction, mother‐infant interaction, neurological development of their babies, postpartum psychological status. Social‐economic status and maternal age were also measured to control their confounding effects.

• Results

• All the outcomes were favourable with continuous support except neurological development of their babies, which showed no difference.

Study 2

• A total of 300 pregnant women in labour in the UK were enrolled in this study at 36 weeks of gestation. Complicated pregnancies including multiples were excluded. The women were allocated in a random order to continuous one‐to‐one support by midwives, or intermittent care by midwives defined as usual care. Main outcomes were intervention (operative deliveries and analgesia) rates in labour and maternal satisfaction at two week postpartum, measured by a structured interview.

• Results

• No difference was found in intervention rates, but satisfaction is slightly better with continuous support.

Study 3

• A total of 6 pregnant women who had continuous support in labour by midwives were followed until 12 month post partum. Detailed semi‐structured interviews (approximately 1.5 hours for each) were conducted by two experienced researchers (a sociologist and an epidemiologist specialized in maternity) before their labour, soon after birth, at 4 weeks, 6 months and 12 months. Their perception towards their childbirth experience was summarised at the end of study period.

• Results

• Women with support in labour showed very positive perception towards their childbirth in general. Their cultural backgrounds and ways of communication between midwives and them seem to greatly affect their perception. All of them had normal vaginal birth without any intervention.

Study 4

• A big national survey was conducted by using telephone interview. A total of 8000 women who have had children were selected by random sampling using national census to ensure geographical, age and cultural background. Interviewers asked them about whether they had support in labour in their latest childbirth, interventions during labourand their perception towards their childbirth.

• Results

• Women with continuous support by caregivers (defined as a person who supports the woman in labour regardless of his/her qualification) have had much fewer interventions in labour and much more positive perception towards their childbirth.

Study 5

• A total of 250 women who had had operative vaginal birth were selected using case notes in a big maternity centre in the UK. Women who had complication prior to deliveries were excluded. Another 250 women who had had normal vaginal birth in the same unit were selected by matching their age, social economic status. They were interviewed on whether they had continuous support in labour in labour ward before the decision of operative birth was made.

• Results• There was no difference in proportion of women who had support in labour.

Integrating results

Meta‐analysis

• Similar studies

• None showed significant impact

• Trend towards positive impact

• A few showed negative impact

77

Shall we integrate them?

An example

Meta‐analysis of effect of skin‐to‐skin contact between mothers and babies on temperature

Review: Physiological changes of newborn babies during skin-to-skin careComparison: 02 Physiological changes (Post SSC - Pre SSC) Outcome: 01 Body temperature

Study During Pre WMD (fixed) Weight WMD (fixed)or sub-category N Mean (SD) N Mean (SD) 95% CI % 95% CI

01 Temperature of the city 10 Celsius degree or lowerLegault 1995 61 36.90(0.30) 61 36.60(0.30) 19.89 0.30 [0.19, 0.41] Fohe 2000 53 37.20(0.20) 53 37.00(0.30) 23.93 0.20 [0.10, 0.30] Bauer 1997 22 35.10(0.70) 22 35.00(0.70) 1.32 0.10 [-0.31, 0.51] Wieland 1995 39 37.10(0.30) 39 37.02(0.28) 13.59 0.08 [-0.05, 0.21] Chiu 2005 39 36.80(0.20) 39 36.70(0.30) 17.61 0.10 [-0.01, 0.21]

Subtotal (95% CI) 214 214 76.33 0.18 [0.13, 0.23]Test for heterogeneity: Chi² = 9.42, df = 4 (P = 0.05), I² = 57.5%Test for overall effect: Z = 6.49 (P < 0.00001)

02 Temperature of the city higher than 10 Celsius degreeLudington 2000 16 36.65(0.34) 16 36.69(0.38) 3.61 -0.04 [-0.29, 0.21] Ludington 2004 11 36.40(0.87) 11 36.33(0.95) 0.39 0.07 [-0.69, 0.83] Clifford 2001 7 36.74(0.28) 7 36.76(0.33) 2.19 -0.02 [-0.34, 0.30] Bosque 1995 8 36.70(0.26) 8 36.80(0.27) 3.34 -0.10 [-0.36, 0.16] Monasterolo 1998 38 36.80(0.30) 38 36.80(0.30) 12.39 0.00 [-0.13, 0.13] Ludington 1991 12 36.46(0.49) 12 36.05(0.61) 1.15 0.41 [-0.03, 0.85] Ibe 2004 13 37.10(0.80) 13 37.10(0.80) 0.60 0.00 [-0.62, 0.62]

Subtotal (95% CI) 105 105 23.67 0.00 [-0.10, 0.10]Test for heterogeneity: Chi² = 4.01, df = 6 (P = 0.68), I² = 0%Test for overall effect: Z = 0.02 (P = 0.98)

Total (95% CI) 319 319 100.00 0.14 [0.09, 0.18]Test for heterogeneity: Chi² = 23.51, df = 11 (P = 0.01), I² = 53.2%Test for overall effect: Z = 5.66 (P < 0.00001)

-1 -0.5 0 0.5 1

Decrease Increase

79

Dealing with heterogeneityI2 statistics and meta‐regression

Meta‐analysis

Heterogeneous

• Test for heterogeneity• Positive

• Content• Heterogeneous

Homogenous

• Test for heterogeneity• Negative

• Content• Homogenous

• Test for heterogeneity• Positive

• Content• Homogenous









-1 -0.5 0 0.5 1

Decrease Increase

82

Studies conducted in cities whose annual average temperature is 10

Celsius degree or lower









-1 -0.5 0 0.5 1

Decrease Increase

83

Studies conducted in cities whose annual average temperature is 10

Celsius degree or lower

Studies conducted in cities whose annual average temperature is warmer than 10 Celsius degree

Meta‐regression

Effect

Variables

Meta‐regression analysis

Effects after skin-to-skin care, compared with before skin-to-skin care

Body temperature

Meta-analysis [degree Celsius]

No of studies 12

Overall results 0.14 [0.09 to 0.18] P<0.001

Test for heterogeneity I2=53.2% P=0.01

Meta-regression analysis coefficient p-value

No of studies 12

Temperature of the city Celsius degree -0.03 0.004

Income of the country high/mid-low 0.34 0.25

Birth weight low/normal -0.64 0.38

Duration of skin-to-skin care duration(min) -0.0004 0.61

85

Meta‐analysis

• Heterogeneity of external validity can be dealt with in meta‐regression analysis

• That of internal validity – not possible

86

Dealing with multi‐arm questionsNetwork meta‐analyses

Network meta‐analyses

Drug A

Drug B

Drug CDrug D

Placebo

Network meta‐analyses

Drug A

Drug B

Drug CDrug D

Placebo

Dealing with publication biasFunnel plot

Funnel Plot

PracticalIf a time allows…

Review Manager 5

• Application to write up Cochrane reviews

• Statistical software to conduct meta‐analysis included

Open RevMan

• Oxygen versus air for neonatal resuscitation

Data Extraction

Intervention (Room Air) Control (100% O2)

Deaths N of Babies Deaths N of Babies

Bajaj 2001 17 107 17 97

Ramji 1993 3 42 4 42

Ramji 2003 26 210 40 221

Saugstad 2003 40 288 61 321

Vento 2001 1 76 2 75

Towards decision makingRintaro Mori MD PhD MSc FRCPCH

Outcomes

• Different study designs for effectiveness of an intervention and patient safety

• RCT for effectiveness

• Observational study for adverse outcomes

• Diagnostic study for diagnostic value

Importance and research findings

• Important outcomes and reported outcomes• not necessarily the same and often different

For Example…

• Many RCT report number of babies with Apgar score less than 7 at 5 minutes but not neurological development

• Many RCT report use of epidural analgesia but not women’s satisfaction with the care in labour

Economic Evidence?

• Integration of economic evidence• How we use limited resources

• Weighing and balancing outcomes• Results often conflicting

• Satisfaction

• Women’s outcome and

• Babies outcome

Economic evidence tries to do this, but often not achievable due to lack of data – we need to use your brain to balance them!

Applicability

• Whether it is applicable to your settings• Not only resources but the context

• An RCT from high income settings

• A cost‐analysis from the US• Can they be extrapolated to your setting?

Yes and No

depending upon the intervention and outcomes

Statistical Significance

• Statistical significance and clinical significance

• P=0.05 (95% CI) – gold standard?

For Example…

• P‐value (95%CI) is a continuous value• Do you intervene if the p‐value=0.049 and not intervene if the p‐value=0.051?

National Collaborating Centre for

Women’s and Children’s Health

P=0.05: The probability is 1 in 20

It is a reasonable standardbut not just be blindly decided by this

Confusion

• Evidence of no difference

• No evidence of difference

For Example…

• An RCT (N=11) reported that CS rate of an intervention:

• RR 4.86 [0.23 to 11.52]

• No evidence of difference

• Does this mean the intervention and the comparator have the same impact?

No!

The RCT is too small to show statistically significant difference

Outcomes

• Effectiveness and adverse outcomes

• Seriousness/rareness/impact of the outcome

• Consequence of the outcomes and interventions

For example

• What are you going to do if:• Satisfaction – RR 1.23 [1.09 – 2.56]

• Vomit – RR 0.92 [0.74 – 0.99]

• CS rate – RR 1.65 [0.78 – 2.37]

• Perinatal mortality – RR 3.86 [0.98 – 11.35]

No AnswerClinical judgement by weighing these outcomes

How you see things…

• Value of the population• Looking at a majority of people, but never forget minorities

Ethics of public health interventions

• Ethical issues• Targeting normal healthy population

For example…

• RCT on routine CS at 37 weeks on all normal healthy women to reduce babies with perinatal asphyxia proved its effectiveness

• Would you do this intervention?

NO!

Caution and clear justificationwhen intervening normal healthy population

E.g. immunisation

Evidence‐based approach upgraded

111

Quantitative

Evidence

Quantitative

Evidence

Health PolicyHealth Policy

Observational Evidence

Qualitative Evidence

Consensus

Decision (economic) analysis

History of Cochrane

• Registry of clinical trials in perinatal medicine in 1970s

• Synthesising trials on the same questions by meta‐analyses – systematic reviews

• Standardised methods and development of archives of systematic reviews – Cochrane Library (CDSR) in 1992

• Pregnancy and Childbirth Group, Neonatal Group, Stroke Group…

• UK Cochrane Centre, Cochrane Canada, US, Scandinavia…

• 30000 conributors from more than 120 countries

• EBM is one of the major inventions in medicine

Cochrane: who we are

• not‐for‐profit

• global, independent network

• 30,000+ contributors

• 120 countriesMission

Our mission is to promote evidence‐informed health decision‐making by producing high‐quality, relevant, accessible systematic reviews and other synthesized

research evidence

An international organisation

www.cochrane.org/contact/centres

Impact Factor

0

2

4

6

8

2007 2008 2009 2010

1. NEJM2. Lancet3. JAMA4. ANN INTERN MED5. PLOS MED6. BMJ7. ANN REV MED8. ARCH INTERN MED9. CMAJ10. Cochrane Database of Systematic Reviews

ChallengesEven beyond…

Evidence‐based health policy

Research SynthesesResearch Syntheses

Health MetricsHealth Metrics

Health policy developed upon research synthesis

• Comparative effectiveness research

Randomised trial and other comparative studies

Systematic review of …

Cost effectiveness analyses of …

Consensus of …

Policy

Health policy developed upon research synthesis

• Comparative effectiveness research

Randomised trial and other comparative studies

Systematic review of …

Cost effectiveness analyses of …

Consensus of …

Policy

The goal set by the cost effectiveness analyses is…

QALYQuality Adjusted Life Years

Cost per QALY

• WHO Guidance• Highly cost‐effective less than GDP per capita

• Cost‐effective between one and three times GDP per capita

• Not cost‐effective more than three times GDP per capita

• UK NICE• 20000 – 30000 GBP







What are the problems?

Problem 1: Goal setting

?QALY/DALY?

Problem 1: Goal setting

?QALY/DALY?

Outcomes of next generations?Outcomes of well being?

Problem 2: Availability and Quality of data• Availability and reliability of empirical data

Do we look for data forever???Dealing with uncertainty

Biased data

Uncertainty

People with flu

Tamiflu

Days lost QALY

Side effects QALY

No Tamiflu

Days lost QALY

Side effects QALY

Based upon meta‐analyses of randomized trials Estimates

Impact of conflict of interest –analysis of 1140 studies

JAMA. 2003;289(4):454-465. doi:10.1001/jama.289.4.454

Health information

Newly developed evidence

Traditionally Developed Evidence

Common Sense

Problem 3: Individualisation

• Wellbeing, needs, demands = subjective matter

• Decision making should be further individualised

• Country‐level versus individual level

• Individualising the information system with new technology

Human security against traditional security

Human Security and Health

Information

Global

Individual

Human Security and Health

Information

Global

Individual

IndividualisationEmpowerment

research synthesis seminar revised - vghtpefdc.weebly.com · Research Synthesis: How to conduct a...

Documents

Transcript of research synthesis seminar revised - vghtpefdc.weebly.com · Research Synthesis: How to conduct a...