Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman,...
-
Upload
clement-hicks -
Category
Documents
-
view
215 -
download
1
Transcript of Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman,...
![Page 1: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/1.jpg)
Course overview, the diagnostic process, and measures of interobserver agreement
Thomas B. Newman, MD, MPH
September 18, 2008
![Page 2: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/2.jpg)
Overview Administrative stuff Overview of the course The diagnostic process Interobserver agreement
– Continuous variables– Categorical variables
• Concordance• Kappa
– Regular– Weighted
![Page 3: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/3.jpg)
Administrative stuff Introductions Basic structure of course
– New material each week in lecture– Read material before lecture if possible– HW on that material due the following week in
section– Exceptions:
• No class October 9• Penultimate class 12/4 – Chapter 12 (Challenges for
EBD) and course review: pass out take-home exam; no HW on Ch 12
• Last lecture 12/11: review of take-home exam Lectures: mixture of PPT and Whiteboard
– How many want paper copies of PPT slides?
![Page 4: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/4.jpg)
SECTIONS
Section assignments: Click ROSTER on Epi 204 website
Section rooms: Click SCHEDULE on website
Faculty will rotate; students, rooms and TA's will be constant for the quarter
![Page 5: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/5.jpg)
Homework Required – key way of learning material Which problems are assigned announced
in SECTION and (later) posted on web Not graded if late, but can still be turned in;
answers on web Use fresh sheets of paper with your name
on each, not syllabus pages, not e-mail. (You can download and word-process if you want, but print a copy unless section leader prefers electronic.)
Will be graded by section leaders and returned the following week
![Page 6: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/6.jpg)
Getting help Classmates, then section leaders, then
faculty Ambiguous/confusing problems – send
e-mail to section leader or me– Unless you indicate otherwise, we will
assume we can cc the whole class when we respond if we think question is of general interest
![Page 7: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/7.jpg)
Textbook
TBN and MAK have almost finished a book, “Evidence-based Diagnosis” (Cambridge University Press, 2009)
Other texts listed in on web Copies of other books in bookstore and on
reserve in the library and available for browsing here
![Page 8: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/8.jpg)
Grading, honor code, etc. Worst HW score dropped; all other HW count
equally 2/3 Homework avg + 1/3 final examination
OR 1/3 Homework avg + 2/3 final examination, whichever is better
Try all problems on your own first; OK to help each other with HW but– Acknowledge help– Write answer in own words
Do not collaborate on final exam Honor code taken seriously
![Page 9: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/9.jpg)
Course overview Diagnosis
– Theory– Inter-rater reliability– Dichotomous tests– Multilevel tests– Studies of tests– Combining tests
Screening and prognostic tests Treatments: randomized trials Alternatives to randomized trials P-values and confidence intervals; Bayes' theorem Clinicians and probability
![Page 10: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/10.jpg)
Diagnostic process Why do we want to assign a name to
this person’s illness? Different reasons lead to different
classification schemes
![Page 11: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/11.jpg)
Examples Acute nephrotic syndrome Acute leukemia Attention deficit disorder Dysuria worth a course of antibiotics SLUBI=Self-limited undiagnosed benign
illness
![Page 12: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/12.jpg)
Simplified Generic Decision Problem
Patient either has the disease or not If D+, net benefit of treatment If D-, better not to treat (“Treat” could include doing more tests)
![Page 13: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/13.jpg)
Simplifying assumptions (often wrong) Test results are dichotomous
– Most tests have more than two possible answers
Disease states are dichotomous– Many diseases occur on a spectrum– There are many kinds of “nondisease”
![Page 14: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/14.jpg)
Evaluating diagnostic tests
Reliability Accuracy Usefulness
Today we do reliability
![Page 15: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/15.jpg)
Types of variables
Categorical– Dichotomous – 2 values– Nominal – no intrinsic ordering – Ordinal – intrinsic ordering
Continuous (infinite number of values) vs Discrete (limited number)
![Page 16: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/16.jpg)
Measuring interobserver agreement for categorical variables
Gallop heard by Observer B
No gallop heard by Observer B
Total, Observer A
Gallop heard by Observer A 20 15 35No gallop heard by Observer A 10 55 65Total, observer B 30 70 100
What is agreement?
![Page 17: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/17.jpg)
Concordance rate
What percent of the time do the 2 observers agree (exactly)
Advantage: easy to understand Disadvantage: may be misleading if
observers agree on prevalence of abnormality
![Page 18: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/18.jpg)
Concordance rate problem
Gallop heard by Observer B
No gallop heard by Observer B
Total, Observer A
Gallop heard by Observer A 0 5 5No gallop heard by Observer A 5 90 95Total, observer B 5 95 100
![Page 19: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/19.jpg)
Unbalanced Disagreement
Lesion # RATER A RATER B
1 S S
2 S S
3 S M
4 S M
5 S M
6 M M
7 M L
8 L L
9 L L
10 L L
BA S M L Total
S 2 2 1 5M 0 0 2 2L 0 0 3 3
Total 2 3 6
What is going on here? Look for lack of balance
above and below diagonal Results when observers
have different thresholds
![Page 20: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/20.jpg)
Definition of Kappa The amount of agreement beyond what
would be expected by chance* Formula:
Practice– Obs = 90%, Exp = 80%, K =– Obs = 70%, Exp = 60%, K =– Obs = 60%, Exp = 70%, K =
*Given the observed marginals
Observed agreement – Expected agreement
1 – Expected agreement
![Page 21: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/21.jpg)
Calculation of Expected Agreement from Marginals
Gallop heard by Observer B
No gallop heard by Observer B
Total, Observer A
Gallop heard by Observer A 20 15 35No gallop heard by Observer A 10 55 65Total, observer B 30 70 100
![Page 22: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/22.jpg)
GCS Eye opening- Observed
Doc #2None To Pain To
CommandSpontaneous Total
None 11 2 0 4 17To Pain 4 1 2 0 7
To Command 0 3 8 3 14Spontaneous 2 1 7 68 78Total 17 7 17 75 116
Emergency Physician #2
![Page 23: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/23.jpg)
GCS Eye Opening: Expected
Doc #2None To Pain To
CommandSpontaneous Total
None 2.5 1 2.5 11 17To Pain 1 0.4 1 4.5 7
To Command 2.1 0.8 2.1 9.1 14Spontaneous 11.4 4.7 11.4 50.4 78Total 17 7 17 75 116
Emergency Physician #2
17 x 78/116 = 1326/116 = 11.4
![Page 24: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/24.jpg)
Why does multiplying row total by column total and dividing by N give you the expected agreement?
![Page 25: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/25.jpg)
Weighted Kappa Weighted kappa
– Linear– Quadratic– Custom
![Page 26: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/26.jpg)
Real-life illustration: Rating of neurological examination Types of weights, Stata illustration.
. tab ex1 ex2
. kap ex1 ex2, w(w)
. kap ex1 ex2, w(w2) (See Appendix 2.1)
![Page 27: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/27.jpg)
What does observed Kappa depend upon?
How well people agree SPECTRUM within classifications
– E.g., re the abnormal ones VERY abnormal?– Difficult cases can be excluded or over-sampled
PREVALENCE of classifications by the various observers (and whether they agree on prevalence)
Chance (random error; people can get lucky/unlucky)
Weighting scheme used
![Page 28: Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 18, 2008.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649eb65503460f94bc046b/html5/thumbnails/28.jpg)
Wireless Internet Access
Key is n2xa8!wr