Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon ...
35
Tracey J. Woodruff, PhD, MPH Professor and Director UCSF Program on ReproducBve Health and the Environment Risk Assessment Specialty Sec1on (RASS) January 14, 2015 Systema(c Reviews, Risk of Bias and Evidence Evalua(on for Characteriza(on of Environmental Chemicals
Transcript of Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon ...
Tracey J. Woodruff, PhD, MPH Professor and Director
UCSF Program on ReproducBve Health and the Environment
Risk Assessment Specialty Sec1on (RASS)
January 14, 2015
Systema(c Reviews, Risk of Bias and Evidence Evalua(on for Characteriza(on of
Environmental Chemicals
3
(CUE workshop has a good cartoon)
Environmental health literature is vast, diverse, and of variable quality
How do we evaluate scien0fic evidence to
make decisions? And shorten the 0me between science and
decision?
5
Clinical sciences have faced and addressed these same challenges
Thank you Lisa Bero
6
What is Evidence Based Medicine?
• Defined as, the “conscientious, explicit, and judicious use of the current best evidence in making decisions about the care of
individual patients” ���
• In general, systems of EBM combine:
(1) existing scientific evidence;
(2) clinical expertise; and
(3) patient values and preferences
to make diagnosis, prognosis and treatment decisions.
7
Grades of Recommendation Assessment, Development
and Evaluation
8
But Evidence Based Medicine
Methodologies Are Not Directly
Transferable to Environmental Science!
8
Naviga1on Guide Methodology
A systemaBc and transparent method to evaluate the quality of evidence and to support evidence-‐based decision making, bridging the gap between clinical
and environmental health
BRIDGING CLINICAL & ENVIRONMENTAL HEALTH
Developed by UCSF’s Program on ReproducBve Health and the Environment in collaboraBon with the NavigaBon Guide Working Group in 2009
Naviga1on Guide Work Group
The Cochrane Collabora1on
Overview of the Methodology
11
“ …systema1c-‐review standards provide an approach that would substanBally strengthen the IRIS process…” NAS 2014
“EPA should consistently use a more systema1c approach to evaluaBng the literature ……….” NAS 2014
12
13
Weight of Evidence
“too vague and is of lidle scienBfic use”
NaBonal Academy of Sciences 2014
Evaluate each evidence stream separately using systemaBc and transparent approaches
Systema1c Review Approach for Each Evidence Stream
“PECO” Statement
SystemaBc search
Select Studies
Extract Data & Data
Analysis
Rate Quality of Evidence
Rate Strength of Evidence Non Human
Data
“PECO” Statement
SystemaBc search
Select Studies
Extract Data & Data
Analysis
Rate Quality of Evidence
Rate the Strength of Evidence
Human Data
Overall Co
nclusio
n
A pre-‐specific analy1c plan (protocol) is developed and applied consistently to the evidence.
Systema1c Review Approach for Each Evidence Stream
“PECO” Statement
SystemaBc search
Select Studies
Extract Data & Data
Analysis
Rate Quality of Evidence
Rate the Strength of Evidence
Overall Co
nclusio
n
A pre-‐specific analy1c plan (protocol) is developed and applied consistently to the evidence.
opula1on xposure omparator utcome
“PECO” Statement
SystemaBc search
Select Studies
Extract Data & Data Analysis
Rate Quality of Evidence
Rate the Strength of Evidence
Rate Quality of Evidence
Rate Strength of Evidence
Rate the Quality and Strength of the Evidence
High Moderate
Low
Sufficient evidence of toxicity
Limited evidence of toxicity
Inadequate evidence of toxicity
Evidence of lack of toxicity
• Risk of Bias
• RaBng quality of evidence
Rate Quality of Evidence
Lam et al. EHP 2014
True Effect 0
True Effect 0
Risk of Bias vs Random Error 1. Bias 2. Random Error
“The commidee notes that assessing the quality of the study is not equivalent to assessing the risk of bias in the study. An assessment of study quality evaluates the extent to which the researchers conducted their research to the highest possible standards and how a study is reported. Risk of bias is related to the internal validity of a study and reflects study-‐design characterisBcs that can introduce a systemaBc error (or deviaBon from the true effect) that might affect the magnitude and even the direcBon of the apparent effect. An assessment of risk of bias is a key element in systemaBc-‐review standards; potenBal biases must be assessed to determine how confidently conclusions can be drawn from the data.” The commidee emphasizes the importance of assessing risk of bias for all study types.
“the commidee emphasizes the need for EPA to assess the “risk of bias” in individual studies.” [for all study types]
NaBonal Academy of Sciences 2014
Risk of Bias & NAS
Low risk
Probably low risk
Probably high risk
High risk
Sequence generation Allocation concealment Blinding Incomplete outcome data Selective reporting
Other bias Conflict of interest
0% 20% 40% 60% 80% 100%
Recruitment strategy Blinding Confounding Exposure assessment Incomplete outcome data Selective reporting Conflict of interest Other bias
Evaluate Risk of Bias
Factors for downgrading/upgrading evidence were derived directly from factors used in GRADE and Cochrane
High
Moderate
Low
Animal evidence • Level of study control • RandomizaBon key factor in default of “high” in GRADE
• Studies find humans more suscepBble to chemical exposures than animals
Human evidence
• Non-‐randomized, but variable quality. Cochrane review finds similar results between RCTs and ObservaBonal studies
Rate the Quality of the Evidence
Factors that DECREASE quality level
1. Risk of bias (study limita1ons)
Possible raBngs: 0=no change; -‐1 or -‐2 downgrade 1 or 2 levels For Human and Animal evidence
2. Indirectness
3. Inconsistency
4. Imprecision
5. Publica1on bias
Factors that INCREASE quality level
1. Large magnitude of effect
2. Dose response
3. Confounding minimizes effect
Possible raBngs: 0=no change; -‐1 or -‐2 downgrade 1 or 2 levels For Human evidence only
NaBonal Academy of Sciences 2014
28
Evidence Stream
Human Non-‐human mammalian
StarBng raBng Moderate High
Downgrade
Risk of Bias 0 -‐1 Indirectness 0 0 Inconsistency 0 0 Imprecision 0 0 PublicaBon bias 0 0
Upgrade
Large magnitude effect 0 N/A Dose response 0 N/A All possible confounding would confirm negaBve result
0 N/A
Grade 0 -‐1
Final ra1ng Moderate Moderate
Summary of Quality of Evidence for PFOA
QUALITY OF EVIDENCE
Moderate
High
Low
STRENGTH OF EVIDENCE (LEVEL OF CERTAINTY REGARDING TOXICITY)
Sufficient evidence of toxicity
Limited evidence of toxicity
Inadequate evidence of toxicity
Evidence of lack of toxicity
CRITERIA: 1. Quality of evidence: 2. What is the direc1on of effect? 3. What is the confidence in the
effect? 4. Are there other compelling
adributes of the data that influence certainty?
Rate the Strength of Evidence
Methodological Needs • Criteria for moving from quality to strength of evidence
• Methods to include all potenBal types of evidence, i.e., assessing chickens, flies and in vitro data
• Improved methods of animal toxicity tesBng – high ROB may be prevalent for key domains
• MechanisBc data is considered under other consideraBons…. Further development needed
• Consider the nature and extent of consensus that is needed for a decision
Conclusion
• Doable • Rigorous, systema1c, transparent and doable • Capacity to evolve with changes in evidence stream
Acknowledgements PFOA Case Study Authors
Acknowledgements Navigation Guide Funders (since 2009)
– USEPA – US Environmental Protection Agency STAR (RD83467801) – National Institute for Environmental Health Sciences (ES018135) – Cal-EPA – Fred Gellert Foundation – Clarence Heller Foundation – New York Community Trust – Forsythia Foundation – Passport Foundation – Johnson Family Foundation – Heinz Endowments – Rose Foundation – Kaiser Permanente – Planned Parenthood Federation of America – UCSF Phillip R Lee Institute for Health Policy Studies
Program on ReproducBve Health and the Environment
Thank you
Sir AusBn Bradford Hill -‐ incompleteness of science … “does not confer upon us a freedom to ignore the knowledge we already have, or to postpone the ac0on that it appears to demand at a given 0me” (Hill 1965).
