Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC...
-
Upload
avice-stevenson -
Category
Documents
-
view
213 -
download
0
Transcript of Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC...
![Page 1: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/1.jpg)
Design and conduct of evaluations of CVD control programs (part I)
Gilles Paradis, MD, MSc, FRCPC
Jennifer O’Loughlin, PhD
McGill University Health Center
Department of Epidemiology and Biostatistics
McGill University
![Page 2: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/2.jpg)
Outline Part I
Why evaluate? What’s evaluation Evaluate what? Scope of evaluation Methodological issues
![Page 3: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/3.jpg)
Why evaluate?
1. AccountabilityReport on the attainment of objectives and use of limited resources
2. ImprovementTreatment, program performance.
3. AdvocacyEnhance programs, build consensus, support coalitions
![Page 4: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/4.jpg)
Why evaluate?
Social responsibility beyond "Primum non nocere"
Many (well established) interventions have been subsequently shown to be useless or harmful
M.I.: Prolonged bed restMagnesiumClass I antiarrhythmicsCa++ channel blockers
Prevention: caroteneHRT (?)
![Page 5: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/5.jpg)
What is evaluation?
Process of systematic data collection or information gathering to shed light on some aspects of an action or intervention
Respond to specific questions regarding a program "Who is being reached by…?"
Support decision making "Which of two alternative strategies is more effective?"
![Page 6: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/6.jpg)
What is evaluation?
Enhance community participation "What are key community concerns?"
Improve the understanding of mechanisms of action"How can I reach low SES populations
with this program?"
Support community mobilization "What do key stakeholders expect from a coalition?"
![Page 7: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/7.jpg)
Evaluate what?
Primary prevention programsReduce exposure to risk factorsDecrease incidence
Secondary preventionPrevent progression among affectedasymptomatic individuals (HBP, …)Screening, case-finding
![Page 8: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/8.jpg)
Individual practiceDiagnostic, preventive, therapeutic
Organizational or community changesStructural (inputs, resources mobilized)Process (quality of services)Outcomes (attainment of objectives)
Tertiary preventionDecrease morbidity, mortality amongsymptomatic individualsImprove QOL, functioning
![Page 9: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/9.jpg)
Scope of Evaluation
Broad approaches
1 - Normative
2 - Evaluative research
![Page 10: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/10.jpg)
Scope of Evaluation
1.1 - Quality of preventive care GOAL: Compare practices to standards of
excellence or criterias EXA: Rules for use of resources
Who gets fasting lipoprotein profiles? Who gets 24 hour BP monitoring? Streptokinase or tpa?
Criterias of quality preventive care Management of HBP, type II diabetes Management of pts with IHD
METHODS: Chart auditsSurveys
1 - Normative
![Page 11: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/11.jpg)
Scope of Evaluation
1.2 - Quality of programs GOAL: Structural: Appropriate use of
resources?Process: Target population
attained?Program implemented
asintended?
Impact: Were objectives achieved?
EXA: HBP screening in worksites Methods: Review of reports, existing databases
Key informant interviewsSurveys
1.3 - Evaluation of (public health) organizations Structure, functioning, planning, etc.
1 - Normative
![Page 12: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/12.jpg)
Scope of Evaluation
Efficacy Effectiveness Efficiency (cost-benefit, cost-
effectiveness) Quality of preventive care (decision
analysis)
2 - Evaluative research
![Page 13: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/13.jpg)
1 - Specification of theoretical model
2 - Design
3 - Measures (what and how)
4 - Biases
5 - Analysis
Methodological issues
![Page 14: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/14.jpg)
1 - Theoretical model
Avoid “Black Box” phenomenon Observe connecting processes between
inputs and outputs Key to understand and improve
interventions Describes how program produces the effect Blueprint for selection of variables, guiding
analysis, interpreting results
Methodological issues
![Page 15: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/15.jpg)
Methodological issues
![Page 16: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/16.jpg)
2 - Design
General modelInitial state Subsequent state
t0 Intervention t1
Effect of intervention, time or other?
Initial state Subsequent state
t0 Intervention t1
Initial state Subsequent state
Methodological issues
![Page 17: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/17.jpg)
Methodological issues
![Page 18: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/18.jpg)
2 - Design
Repeat cross-sectional surveys Cohort RCT (Case-control)
Methodological issues
![Page 19: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/19.jpg)
2 - Design
Cohort Individual behavior change Non-anonymous participation Attrition related to behavior
evaluated Repeat testing, co-intervention Maturation, aging More long-term residents 1-
Repeat C / S Community-wide prevalence Anonymous
Less of a problem Cross-contamination 1-
Methodological issues
![Page 20: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/20.jpg)
Methodological issues
![Page 21: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/21.jpg)
2 - Design
RCT Unbiased allocation Similar distribution of R.F. (known or unknown)
to groups Comparability of groups Validity of statistical tests Feasability, costs Other options to minimize biases (matching,
stratification, …)
Methodological issues
![Page 22: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/22.jpg)
Methodological issues
![Page 23: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/23.jpg)
3 - Measures
3.1 - What? Mortality, morbidity Q O L Risk factors Behaviors Physical and social environments
Proximal impact easier to measure than distal
Methodological issues
![Page 24: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/24.jpg)
3 - Measures
3.2 - How?ReliabilityValidity
Self-reported behaviors Social desirability Pre-testing instruments Objective measures / Gold standard Environmental measures(shelf space, no-smoking
signs, …) Surrogate reports from next of kin Bogus measurements
Methodological issues
![Page 25: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/25.jpg)
4 - Biases
“Distortion in the estimate of effect of an exposure”
due to Selection of subjects How information is collected Confounding
Methodological issues
![Page 26: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/26.jpg)
4 - Biases
Community programs particularly prone to biases
Random allocation is rare Limited # of clusters Important differences between groups
(absolute and secular trends) Multiple co-interventions Blinding is impossible
Methodological issues
![Page 27: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/27.jpg)
Solutions:
Matching:
# of pairs
# measurements
Methodological issues
![Page 28: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/28.jpg)
Methodological issues
![Page 29: Design and conduct of evaluations of CVD control programs (part I) Gilles Paradis, MD, MSc, FRCPC Jennifer O’Loughlin, PhD McGill University Health Center.](https://reader035.fdocuments.in/reader035/viewer/2022062713/56649ced5503460f949baa6d/html5/thumbnails/29.jpg)
5 - Analysis
Effects measured at the individual level but allocation and intervention are at the community level
1- High intra-class correlationsBiased standard error at the individual level ( false -positive results)Standard error must be computed at the community level
• requires N• adjustment for sampling procedures• # data collection
Methodological issues