Lecture 4 Study design and bias in screening and diagnostic tests
description
Transcript of Lecture 4 Study design and bias in screening and diagnostic tests
![Page 1: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/1.jpg)
1
Lecture 4Study design and bias in
screening and diagnostic tests
• Sources of bias :– spectrum effects/subgroup analyses – verification/workup bias– information (review) bias
• Critical assessment of studies:– e.g., STARD criteria
![Page 2: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/2.jpg)
2
Bias
• What is it?– Bias in a measurement vs bias in the result of a
study
• Selection vs information bias• What does it mean in studies of screening
and diagnostic tests?• Difference between bias and effect
modification?
![Page 3: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/3.jpg)
3
Reducing bias
• Studies of diagnostic tests give variable results
• Biassed studies generally overestimate sensitivity/specificity
• STARD criteria proposed to improve quality of these studies
![Page 4: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/4.jpg)
4
Spectrum effect: bias or modification?
• Sensitivity and specificity are not innate characteristics of a test, but vary by study population – e.g., by age, sex, comorbidity– e.g, exercise stress testing: worse performance
in women than men– Study population should be representative of
population in which test will be used
![Page 5: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/5.jpg)
5
Design implications
• Investigate test performance in sub-groups
• Report characteristics of study population
![Page 6: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/6.jpg)
6
Verification/work-up bias
• Results of test affect intensity of subsequent investigation– e.g., risky or expensive follow-up
• Selection or information bias?
• E.g. Exercise stress test and angiography– effects?– solutions?
![Page 7: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/7.jpg)
7
Example of verification/work-up bias
• VQ (ventilation/perfusion scanning to detect pulmonary embolism – Positive scan -> angiography– Studies with selective referral of patients:
sensitivity = 58%– Study (PIOPED) with prospective investigation
of all patients: sensitivity = 41%
![Page 8: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/8.jpg)
8
Information/review bias
• Examples:– Diagnosis is not blind to test result– Diagnosis is made with access to other clinical
information – Knowledge of results of follow-up used in
interpretation of screening test
• Effects?• Solutions?
– (NB: raw test performance vs “real-world” situation)
![Page 9: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/9.jpg)
9
Other sources of bias
• Indeterminate test results:– How do they affect results?– Solutions?
• Context:– Interpretation varies with changes in disease
prevalence
• Criteria for positivity– Technical advances, operator experience
![Page 10: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/10.jpg)
10
Optimal design
• Cohort vs case-control?
• Prospective cohort with blind evaluation
• Case-control:– Sources of bias?
![Page 11: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/11.jpg)
11
Example for discussion
• Seniors in emergency department (ED):– High risk of functional decline, death etc.– Needs usually not recognized at ED visit
• Objective: Development and validation of tool to identify “high-risk” seniors in ED (need more careful assessment and follow-up)
• Methods?
![Page 12: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/12.jpg)
12
RESULTS: ISAR development
Adverse health outcome defined as any of following during 6 months after ED visit
• >10% ADL decline
• Death
• Institutionalization
![Page 13: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/13.jpg)
13
0
10
20
30
40
50
60
ADL decline Death Institution Any
%
DischargedAdmitted
Adverse outcome by disposition
![Page 14: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/14.jpg)
14
Scale development
• Selection of items that predicted all adverse health events
• Multiple logistic regression - “best subsets” analysis
• Review of candidate scales with clinicians to select clinically relevant scale
![Page 15: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/15.jpg)
15
Identification of Seniors At Risk (ISAR)
1. Before the illness or injury that brought you to the Emergency, did you need someone to help you on a regular basis? (yes)
2. Since the illness or injury that brought you to the Emergency, have you needed more help than usual to take care of yourself? (yes)
3. Have you been hospitalized for one or more nights during the past 6 months (excluding a stay in the Emergency Department)? (yes)
4. In general, do you see well? (no)
5. In general, do you have serious problems with your memory? (yes)
6. Do you take more than three different medications every day? (yes)
Scoring: 0 - 6 (positive score shown in parentheses)
![Page 16: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/16.jpg)
16
0
20
40
60
80
0 1 2 3 4 5-6
ISAR SCORE
%
DischargedAdmitted
Any adverse outcome by ISAR score and disposition
![Page 17: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/17.jpg)
17
Predictive validity of ISAR scale
• AUC and 95% CI– Overall (n=1673): 0.71 (0.68 – 0.74)– Admitted to hospital (n=509): 0.66 (0.61 –
0.71) – Discharged (n= 1159): 0.70 (0.66 – 0.74)– Similar results by informant (patient vs proxy)
• Next steps?
![Page 18: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/18.jpg)
18
Second study
• Multi-site randomized controlled trial of a 2-step intervention using ISAR + nurse assessment/referral
• Study 2 population had lower % ISAR +ve than study 1 population – implications for sensitivity, specificity, AUC,
LR, DOR?
![Page 19: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/19.jpg)
19
Area Under the curve (AUC) for concurrent validity criteria
Detection of depression
at baseline
Study 2
OARS: Study 1
Severe functional
impairment
OARS: Study 2
SMAF: Study 1
AUC (95% confidence interval)
0.5 0.6 0.7 0.8 0.9 1.0
![Page 20: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/20.jpg)
20
Area Under the Curve(AUC) for predictive validation criteriaamong patients discharged from ED
Increase in depressivesymptoms
Study 2
10+ community healthcenter visits/5 months
Study 2
11+ hospital days/ 5 months
Study 1
Study 2
2+ ED visits/ 5 months
Study 1
Study 2
Adverse health outcome
Study 1
AUC (95% confidence interval)
0.5 0.6 0.7 0.8 0.9 1.0
![Page 21: Lecture 4 Study design and bias in screening and diagnostic tests](https://reader035.fdocuments.in/reader035/viewer/2022070405/56814015550346895dab5e3e/html5/thumbnails/21.jpg)
21
Other predictive measures in elderly• Pra screening tool (widely used in US HMOs):
– AUC values of 0.61 - 0.71 for prediction of hospital utilization or functional decline (Coleman, 1998)
• Hospital Admission Risk Profile (HARP)– AUC of 0.65 for prediction of nursing home
admission (Sager, 1996)• Comorbidity indices (diagnosis and medication-based
measures from administrative data):– AUC values of 0.58-0.60 for emergency
hospitalization (Schneeweiss, 2001)