MAKING SENSE OF DIAGNOSTIC META-ANALYSIS A VISUAL- GRAPHICAL FRAMEWORK Ben A. Dwamena, MD....
-
date post
22-Dec-2015 -
Category
Documents
-
view
222 -
download
0
Transcript of MAKING SENSE OF DIAGNOSTIC META-ANALYSIS A VISUAL- GRAPHICAL FRAMEWORK Ben A. Dwamena, MD....
MAKING SENSE OF DIAGNOSTIC META-ANALYSIS A VISUAL-GRAPHICAL FRAMEWORK
Ben A. Dwamena, MD. Department of Radiology, University of Michigan Medical School/ Nuclear Medicine Service (115), VA Ann Arbor Health Care System, Ann Arbor.
Francesca C. Dwamena, MD. Department of Medicine, Division of General Internal Medicine, Michigan State University College of Human Medicine, East Lansing.
EDUCATIONAL GOALS AND OBJECTIVES
PROVIDE A VISUAL/GRAPHIC FRAMEWORK FOR DIAGNOSTIC META-ANALYSIS THROUGH THE USE OF – Flow Charts – Contingency Tables – Forest Plots– Funnel Graphs– Scatter Diagrams– Summary Receiver Operator Characteristic Curves– Etc
DIAGNOSTIC META-ANALYSES
Improve quality of future primary studies
by identifying methodological deficiencies
Identify reasons for variation in reported
results
Generate valid summary estimates of
diagnostic performance
VISUAL DISPLAYS/GRAPHS
Provide more user-friendly summaries of large quantitative data sets
Preliminary data exploration before definite data synthesis
Clarify difficult statistical concepts and interpretation
GENERAL ARCHITECTURE
Formulate Question
Develop Search Strategy and Retrieve
Articles
Select Eligible Studies and Assess Quality
Extract Data and Calculate Individual
Summary Measures
Choose Model for Pooling
Investigate Heterogeneity and Biases
DATA SOURCES
VISUAL/GRAPHIC DISPLAYS BASED ON
Dwamena BA, Sonnad SS, et al.
Metastases From Non-small Cell Lung
Cancer: Mediastinal Staging In The
1990s- A Meta-analytic Comparison Of
PET and CT.Radiology 1999; 213:530-
36.
Published Work of Other Investigators in
the Field Based on Either Original or
Simulated Data
RESEARCH QUESTION
GENERAL EXAMPLE: How Accurate Is a Sign, Symptom, or Diagnostic Test in Predicting the True Diagnostic Category of a Patient?
RELEVANT QUESTION: Addresses Population or Patient Group, Diagnostic Intervention, Disease of Interest
FINDING RELEVANT STUDIES
SEARCH FOR EXISTING REVIEWS
FIND PUBLISHED PRIMARY STUDIES
Break down research question into components
Use appropriate synonyms
Use electronic databases, hand searching,etc
LOOK FOR UNPUBLISHED PRIMARY
STUDIES
(write to experts, search registries for
completed/ongoing trials)
BREAKING QUESTION DOWN INTO
COMPONENTS “What is the accuracy of fecal occult blood test for detection of colorectal
cancer? “ may be represented by a VENN DIAGRAM:
RECOMMENDED SEARCH STRATEGY REGARDING TEST
PERFORMANCE
Deville WL et. al. BMC Medical Research Methodology 2002, 2:9-22
QUALITY CITERIA
PATIENT SELECTION: Consecutive vs. Non-consecutive or convenience
sample
SPECTRUM: Clinically relevant population versus case-control
REFERENCE STANDARD: Full vs. Partial reporting of cut-off value
DIAGNOSTIC TEST: Full vs. partial reporting of cut-off value
DATA COLLECTION: Prospective versus Retrospective versus unknown
DETAILS OF POPULATION: Sufficient versus Insufficient
VERIFICATION: Complete versus different reference tests versus
incomplete
INTERPRETATION OF RESULTS: Blinded versus Unblinded
METHODOLOGICAL STANDARDS
Quality of Each Selected Paper Should Be Assessed
Independently by at Least Two Reviewers.
Chance-adjusted Agreement Should Be Reported
and Disagreements Solved by Consensus or
Arbitration.
To Improve Agreement, Reviewers Should Pilot
Their Quality Assessment Tools in a Subset of
Included Studies or Studies Evaluating a Different
Diagnostic Test
METHODOLOGICAL STANDARDS
FLOW CHART OF STUDY RETRIEVAL AND
SELECTION
DATA EXTRACTION
Info About the Participants Included in the Study, Time of Data Collection and the Testing Procedures.
The Cut-off Point Used in Dichotomous Testing Reasons and the Number of Participants Excluded Because of Indeterminate Results or Unfeasibility.
Extracted Information May Be Used in Subgroup Analyses and Statistical Pooling.
DATA EXTRACTION
Multiple Reviewers Should Independently Extract the Required Information.
Obtain Data Construct the Diagnostic 2 × 2 Table: Absolute Numbers in the Four Cells Are Needed.
Obtain Totals 'Diseased' and 'Non-diseased' to Calculate Prior Probability (Pre-test Probability) From Recalculated Sensitivity, Specificity, Likelihood Ratios, Predictive Values
CONTINGENCY TABLE FOR EXTRACTION OF TEST
DATA
DIAGNOSTIC VS. TREATMENT TRIAL
True Positives =Experimental Group With the Monitored Outcome Present (a).
False Positives = Control Group With the Outcome Present (b).
False Negatives=experimental Group With the Outcome Absent (c).
True Negatives Are the Control Group With the Outcome Absent (d).
DIAGNOSTIC VS. TREATMENT TRIAL
Relative risk in experimental group {[a/(a +
c)]/[b/(b+ d)]} =Likelihood Ratio for a
Positive Test.
Relative Risk in Control Group = Likelihood
Ratio for a Negative Test.
The Expression for the Odds Ratio (OR) Is (a
x d)/(b x c).
CONTINGENCY DATA FOR NSCLC PET STUDY
CHOICE OF MODEL AND INDEX FOR
POOLING OF TEST PERFORMANCE
SEARCHING FOR THRESHOLD EFFECT
Test for the Presence of Threshold Effect
Between Studies by Calculating a Spearman
Correlation Coefficient Between Sensitivity
and Specificity of All Included Studies
A Spearman Correlation of < -0.6, Suggests
Evidence of Interdependence of Sensitivity
and Specificity , and SROC Curves Should Be
Constructed or ROC Curves Can Be Pooled
SEARCHING FOR HOMOGENEITY
Perform Chi-square or Fisher’s Test for Small Number
Studies.
If Sensitivity and Specificity Are Homogeneous, and Show
No Threshold Effect, They Can Be Pooled by Fixed Effect
Model.
If Heterogeneity Is Present, Restrict the Analysis to a
Qualitative Overview; Pool Data From Homogeneous Sub-
groups; Use Random Effect Model.
SROC PROCEDURE
1 SCATTER PLOT TPR VS. FPR
Visualization of range of results from primary studies
2 REGRESSION OF D ON S
Straight lines fitted to estimate (a) best fit to the data (b) remove effect of possible relationship between results and positivity threshold
3 BACKTRANSFORMATION OF REGRESSION TO CONVENTIONAL AXES
Presentation of combined results into a single ROC curve
LINEAR REGRESSION ANALYSIS
Logit transformations of the TP rate
(sensitivity) and FP rate (1 - specificity).
D=ln(DOR) =logit(TPR) – logit(FPR)
Differences in logit transformations, D,
regressed on sums of logit transformations, S.
S=logit(TPR)+logit(FPR)
Logit(TPR)=natural log odds of a TP result and
logit(FPR) =natural log of the odds of a FP test
result.
LINEAR REGRESSION MODELS
ORDINARY LEAST SQUARES METHODStudies are weighted equally
WEIGHTED LEAST SQUARES METHODWeighted by the inverse variance weights of the diagnostic odds ratio, or simply the sample size
ROBUST-RESISTANT METHODMinimizes the influence of outliers
LINEAR REGRESSION PLOT
SUMMARY ROC CURVE
Back transformation of logistic regression to conventional axes of sensitivity [TPR] vs. (1 – specificity) [FPR]) with the equation
TPR = 1/{1 + exp[- a/(1 - b )]} [(1 - FPR)/(FPR)](1 + b )/(1 - b ).
Slope (b) and intercept (a) are obtained from the linear regression analyses
SUMMARY ROC CURVE
FOREST PLOT OF STUDY-SPECIFIC AND SUMMARY SENSITIVITY AND
SPECIFICITY
FIXED EFFECTS META-ANALYSIS
Assumes Same Diagnostic Accuracy in All Studies
Variation in Sensitivity and Specificity From Published Reports Due to
Random Error/chanceThreshold Variation
FIXED EFFECTS META-ANALYSIS
RANDOM EFFECTS META-ANALYSIS
Assumes Diagnostic Accuracy Varies From Study to Study
Variation in of Reported Accuracy Estimates Are Randomly Distributed About Some Central Value Represented by SROC.
Variation Due to Stage of Disease, Clinical Presentation, Prevalence of Disease, Study Design Etc.
RANDOM EFFECTS META-ANALYSIS
WEIGHTED HISTOGRAM BREAST CANCER DATA
DEALING WITH HETEROGENEITY
Repeat Analysis Analysis After Excluding Outliers
Conduct Analysis With Predefined Subgroups.
Use Analysis of Variance With the Lndor As
Dependent Variable and Categorical Variables for
Subgroups As Factors to Look for Differences Among
Subgroups;
Construct Multivariate Models to Search for the
Independent Effect of Study Characteristics
GALBRAITH PLOT
Standardized Log-odds Ratio Plotted Against the Reciprocal of the Standard Error.
Small Studies/less Precise Results Appear on the Left Side and the Largest Trials on the Right End .
A Regression Line , Through the Origin, Represents the Overall Log-odds Ratio.
Lines +/- 2 Above Regression Line, Represent the 95 Per Cent Boundaries of the Overall Log-odds Ratio.
The Majority of Results Should Lie in This Area in the Absence of Heterogeneity.
GALBRAITH PLOT
FUNNEL DIAGRAM
A Funnel Diagram (A.K.A. Funnel Plot, Funnel Graph)
Is a Special Type of Scatter Plot With an Estimate
Sample Size on One Axis and Effect-size Estimate
on the Other Axis.
Based on the Well Known Statistical Principle That
Sampling Error Decreases As Sample Size Increases.
Used to Search for Publication Bias and to Test
Whether All Studies Come From a Single Population.
SIMULATED FUNNEL PLOTS FOR EXPLORING
PUBLICATION BIAS
FUNNEL PLOT REGRESSION
EGGER’S REGRESSION METHOD FOR DETECTING
PUBLICATION BIAS
FUNNEL PLOT OF NSCLC PET DATA
NORMAL QUANTILE PLOTS
NORMAL QUANTILE PLOTS
NORMAL QUANTILE PLOT OF
NSCLC PET DATA
NORMAL QUANTILE PLOT OF AXILLARY BREAST CANCER PET DATA
ODDS RATIO SUBGROUP ANALYSIS
SROC SUBGROUP ANALYSIS
SOFTWARE
METATEST (Dr Lau, NEMC, Boston)
SROC Curve Analysis
METAWIN (Sinauer Associates, Sunderland, MA)
Scatter, Funnel, Normal Quantile, Forest, Weight
Histogram, Radial (Galbraith) and Cummulative Meta-
analysis Plots
STATSDIRECT(StatsDirect Ltd, Herts, UK)
Forest, Funnel and L’abbe Plots