Meta-analysisPartini P. TrihonoNastiti Kaswandani

Child Health Department Faculty of Medicine University of Indonesia

BACKGROUNDOver 2,000,000 articles published annuallyOver 20,000 journalsOver 4,400 pages devoted to 11,000 articles in BMJ and NEJM

So how should we go about summarizing medical information?

Types of Reviews

NARRATIVESYSTEMATICQUESTIONBroadFocusedSOURCES / SEARCHUsually unspecified; possibly biasedComprehensive; explicitSELECTIONUnspecified; possibly biasedCriterion-based; uniformly appliedAPPRAISALVariableRigorousSYNTHESISQualitative onlyEitherINFERENCESeldom evidence-basedUsually evidence-based

What is a Systematic Review?A systematic review is an overview of primary studies which contains an explicit statement of objectives, materials and methods and has been conducted according to explicit and reproducible methodology.Two types: Qualitative (non-statistical) and Quantitative (statistical)A quantitative systematic review is also referred to as a meta-analysis.

Review, Systematic review & Meta-analysisSystematicreviewMeta-analysisReviewquantitativepooling of results of individual studies

What is meta analysis?Quantitative approach for systematically combining results of previous research to arrive at conclusions about the body of research.

What does it mean?Quantitative : numbersSystematic : methodicalCombining : putting togetherprevious research: what's already doneConclusions : new knowledgeConsidered as a true investigation observational - retrospective study

Most meta-analyses are developed for clinical trials, but there have been increasing interest in observational studies, esp. cohort studies & diagnostic tests

Hierarchy of studies

Meta-analysis: anatomy & physiology Introduction:Background / JustificationResearch question(s) Objectives Hypothesis Methods:A. Eligibility criteria:inclusion (e.g. RCT only, yearof publication, published orunpublished as well, diagnosis, age group, language, etcexclusion

Methods (cont)B. Evaluation of articles:1. Check for eligibility (Abstract), 2. Evaluating the quality of articles using pre- determined criteria usu. by 2 indep reviewer consensus if necessary score system: good -> poor weighting (large vs small studies)C. AbstractionD. Combining results: statistical technique usedE. Reporting the results

Steps in meta-analysis:IdentificationSelectionAbstractionAnalysisAll should refer research questions!!

A. IdentificationFind published relevant articles: On-line journals, on-line databasesHand-searching: textbooks, printed journals, Index MedicusPlus unpublished materials: theses, dissertations, personal communication (peer group), etc

B. SelectionDesign: e.g. randomization (for clinical trial)Do they included needed information?Patients characteristics: diagnosis (incl. severity) , sex, age, group, race, nutritional status, etcYear of studyDosageLanguageMinimum sample sizeStudy setting: clinic, ICU, field study, etc.

C. AbstractionCompile relevant dataMay need information from author(s)Summarize for analysisWeighted scoring: studies with large n & more definite design weigh more

D. Analysis Many techniquesStill controversialFocus on variabilityFixed effects model only intra-study, ignore interstudy var narrower CIRandom effects model considers also inter-study var broader CIBayesian statisticians (less popular): prior probability even broader CI

Unless the studies are very heterogeneous,results of fixed and random effects model usually similar

Test of homogeneity (x2, graphs)If homogenous: use fix effects modelIf heterogeneous: use random effects model

Study results: Usually presented in graphs

For nominal outcomes: Presented as OR for individual study with corresponding 95% CI Represented by square, the area = relative weighted value Summarized as combined OR with CI as diamond Vertical line: OR = 1 (no effect) Vertical dotted line: if CI crosses this line: means homogenous; otherwise: heterogeneous

Meta-analysis: Example of graph presentationof RCTs with nominal outcome

Study results:

For numeric outcomes: Presented as mean difference for individual study with corresponding 95% CI Represented by square, the area = relative weighted value Summarized as combined mean difference with CI as diamond Vertical line: mean diff = 0 (no effect) Vertical dotted line: if CI crosses this line: means homogenous; otherwise: heterogeneous

Meta-analysis: Example of graph presentationof RCTs with numerical outcome (xe-xc)

Forest plotThe label tells you what the comparison and outcome of interest are

Forest plotEach study has an ID

Forest plotThe data for each trial are here, divided into experimental and control groups

Forest plotThis is the % weight given to each study in the pooled analysis

Forest plotResults from each trial are also given numerically

Forest plotThe size of the block for each study is proportional to the % weight. The horizontal line represents the confidence interval.

Forest plotThe label above the graph tells you which statistic has been used

Forest plotTake care to read the labels things to the left do not always mean the treatment is better than the control

Forest plotThe pooled analysis is given a diamond shape The widest bit is located on the point estimate and the horizontal width in the confidence interval. Does the confidence interval cross the line of no effect?

Forest plotResults of the pooled analysis are also given numerically

Forest plotTest for heterogeneityThe result of a Z test with a p value are also given for the overall effect

Publication bias

Investigators tend to submit studies with positive than negative resultsEditors tend to accept studies with positive than negative resultsPositive results: submit to international journalsNegative results: submit to local journalsMany negative results are product of studies withsmall sample sizeInference by manufacturers!

Funnel Plot: what and how to readPlots the effect size against the sample size of the studyTo study a funnel plot, look at its LOWER LEFT corner, thats where negative or null studies are locatedIf EMPTY, this indicates PUBLICATION BIASNote that here, the plot fits in a funnel, and that the left corner is not all that empty, but we cannot rule out publication bias

Evaluating meta-analysis

ValidityImportanceApplicability

Critical Appraisal of a Meta-analysis1. Validity

GUIDECOMMENTSDid the meta-analysis address a focused question?Is the main question clear from the title or abstract?Were the criteria used to select articles for inclusion appropriate?Specify the subjects, exposures, and outcomes? Specify the methodological standards used to select studies? Is it likely that important relevant studies were missed?Authors conducted a thorough search? Identify unpublished studies?

Critical Appraisal of a Meta-analysis1. Validity

GUIDECOMMENTSWas the validity of the included studies appraised?Were the assessments of studies reproducible?Did two or more people participate to select studies? Were the results similar from study to study?Tests of Homogeneity?

Critical Appraisal of a Meta-analysis2. Importance

GUIDECOMMENTSWhat are the overall results of the meta-analysis?Weighted studies according to their size? RR and the absolute risk reduction? Likelihood Ratios? ORs? Effect Size?How precise were the results?Confidence interval around the estimate?

Critical Appraisal of a Meta-analysis3. Applicability

GUIDECOMMENTSCan the results be applied to my patient care?Similarity of your patients and reviewed patientsYour local settingWere all clinically important outcomes considered?Are the benefits worth the harm and costs?Patients preference

1. ValidityDid the authors ask focused clinical questions?Were the criteria used in the selection of articles appropriate?Is it unlikely that important studies missed?Was the validity of the included studies appraised?Were the assessment of studies reproducible?Were the results similar from study to study?

2. ImportanceWere the overall results clinically important?NNT can be determined by formulas or tables

3. ApplicabilitySimilarity of your patients and reviewed patients Your local settingPatients preference

INTEGRATIVE LITERATUREOverview (review article): unsystematicSystematic review: in gathering, evaluating, presenting evidenceno formal statistical methodsMeta-analysis: systematic review plus formal statistical analysis

Review articleSystematic reviewMeta analysisINTEGRATIVE LITERATURE

Review articleSystematic reviewMeta analysisINTEGRATIVE LITERATURE

Review article = literature review, overview, state of the art review

Non-systematic: in gathering relevant studies no sufficient appraisalsProne for severe bias authors tend to cite studies that support their opinion example: Linus Pauling and Vit CStill valuable in some areas of study, e.g. clinical & lab descriptions of diseases, pathophysiology & other basic sciences

Systematic review:Systematic in: Gathering relevant articles Appraising the articlesNo formal statistical analysisMeta-analysis:Systematic review with formalstatistical analysis

Why meta-analysis?To quantitatively summarize estimate from previous studies (i.e. resolve controversies) Helps guide for further research

Meta-analysis = Statistical technique to systematically and quantitatively combined combinable results of two or more studies Considered as a true investigation: observational - retrospective study

Most meta-analyses are developed for clinical trials, but there have been increasing interest in observational studies, esp. cohort studies & diagnostic tests

Basic study designsObservationalCase reportsCase seriesCross-sectional studiesCase-control studiesCohort studiesMeta-analysisInterventionalLab experimentsClinical trialsField trials

Test of sensitivity:Was the result robust?Random & fixed effects: similarExclusion of low weighted studies: similar? Publication bias: larger sample size gave smaller effect sizeExclusion of special cases, e.g. incomplete follow-up

Meta-analysis: Advantages & disadvantages

Advantages:Combined effect size: more definitiveEnables subgroup analysisBackground for further studiesDisadvantages:Statistical controversiesPublication biasDisadvantages of secondary dataTechnical limitations: Details of studies rarely identical

********