Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical...
-
Upload
eunice-bradford -
Category
Documents
-
view
216 -
download
0
Transcript of Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical...
![Page 1: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/1.jpg)
Natural Language Processing for Biosurveillance
Wendy W. Chapman, PhD
Center for Biomedical Informatics
University of Pittsburgh
![Page 2: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/2.jpg)
Overview
• Motivation for NLP in Biosurveillance
• Evaluation of NLP in Biosurveillance– How well does NLP work in this domain?– Are NLP applications good enough to use?
• Conclusion
![Page 3: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/3.jpg)
What is Biosurveillance andWhy is NLP Needed?
![Page 4: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/4.jpg)
Biosurveillance
• Threat of bioterrorist attacks– October 2002 Anthrax attacks
• Threat of infectious disease outbreaks – Influenza
– Sudden Acute Respiratory Syndrome
• Early detection of outbreaks can save lives
• Outbreak Detection– Electronically monitor data that may indicate outbreak
– Trigger alarm if actual counts exceed expected counts
![Page 5: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/5.jpg)
Emergency Department: Frontline of Clinical Medicine
Whatis themattertoday?
Electronic Admit Data• Free-text chief complaint• Coded Admit diagnosis (rare)• Demographic Information
Triage Nurse/Clerk Physician
Electronic Records• ED Report• Radiology Reports• Laboratory Reports
Electronic Admit Data• Free-text chief complaint• Coded Admit diagnosis (rare)• Demographic Information
Electronic Records• ED Report• Radiology Reports• Laboratory Reports
![Page 6: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/6.jpg)
RODS System
Emergency Department
Admission Records fromEmergency Departments
Emergency Department
Emergency Department
Graphs and MapsRODS System
Database
DetectionAlgorithms
NLP Applications
WebServer
GeographicInformation
System
Preprocessor
![Page 7: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/7.jpg)
![Page 8: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/8.jpg)
![Page 9: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/9.jpg)
Possible Input to RODS
Pneumonia Cases
RespiratoryFinding
FeverPneumonia on
Chest X-ray
Increased WBCCount
Probability ofPneumonia
yes
yes yes
yes
99.5%
![Page 10: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/10.jpg)
How To Get Values for the Variables
• ED physicians input coded variables for all concerning diseases/syndromes
• NLP application automatically extract values from textual medical records
Our research has focused on extracting variablesand their values from textual medical records
![Page 11: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/11.jpg)
Evaluation of NLP in Biosurveillance
![Page 12: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/12.jpg)
Goals of Evaluation of NLP in Biosurveillance
• How well does NLP work?– Technical accuracy
• Ability of an NLP application to determine the values of predefined variables from text
– Diagnostic accuracy• Ability of an NLP application to diagnose patients
– Outcome efficacy• Ability of an NLP application to detect an outbreak
• Are NLP applications good enough to use?– Feasibility of using NLP for biosurveillance
![Page 13: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/13.jpg)
NLPSystem
• Respiratory Fx: yes• Fever: yes• Positive CXR: no• Increased WBC: no
Medical Record
Technical Accuracy
Number of patients with
PneumoniaOutcome Efficacy
RespiratoryFinding
FeverPneumonia on
Chest X-ray
Increased WBCCount
Probability ofPneumonia
Diagnostic Accuracy
![Page 14: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/14.jpg)
Technical AccuracyCan we accurately identify variables from text?
• Does measure NLP application’s ability to identify findings, syndromes, and diseases from text
• Does not measure whether or not patient really has finding, syndrome, or disease
NLPApplication
Variable values from Reference
Standard
Variable Values from NLP
NLP Application Performance
Compare
Text
Reference Standard
![Page 15: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/15.jpg)
Chief Complaints
![Page 16: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/16.jpg)
Extract Findings from Chief Complaints
Input Data Variable
Free-text chief complaint
NLPApplication
Specific Symptom/Finding
• Diarrhea• Vomiting• Fever
![Page 17: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/17.jpg)
Results
Diarrhea Vomiting Fever
Sensitivity 1.0 1.0 1.0
Specificity 1.0 1.0 1.0
PPV 1.0 1.0 1.0
NPV 1.0 1.0 1.0
![Page 18: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/18.jpg)
Classify Chief Complaints into General Syndromic Categories
“cough wheezing”
“SOB fever”
Respiratory
Respiratory
“vomiting abd pain”
“N/V/D”
Gastrointestinal
Gastrointestinal
Input Data Variable
Free-text chief complaint
Syndromicpresentation
NLPApplication
![Page 19: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/19.jpg)
Chief Complaints to Syndromes
Two Text Processing Syndromic Classifiers
• Naïve Bayesian text classifier (CoCo)*
• Natural language processor (M+)**
Methods• Task: classify chief complaints into one of 8 syndromic
representations
• Gold standard: physician classifications
• Outcome measure: area under the ROC curve (AUC)
* Olszewski RT. Bayesian classification of triage diagnoses for the early detection of epidemics. In: Recent Advances in Artificial Intelligence: Proceedings of the Sixteenth International FLAIRS Conference;2003:412-416.** Chapman WW, Christensen L, Wagner MM, Haug PJ, Ivanov O, Dowling JN, et al. Classifying free-text triage chief complaints into syndromic categories with natural language processing. AI in Med 2003;(in press).
![Page 20: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/20.jpg)
Results: Chief Complaints to Syndromes
* There were no Botulinic test cases for M+
0
0.2
0.4
0.6
0.8
1
Syndrome
AU
C M+
NBCoCo
![Page 21: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/21.jpg)
Chest Radiograph Reports
![Page 22: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/22.jpg)
Evidence for Bacterial Pneumonia
Detection of Chest x-ray reports consistent with pneumonia
Sym-Text
U-KS P-KS
Sensitivity 0.95 0.87 0.85
Specificity 0.85 0.70 0.96
PVP 0.78 0.77 0.83
NPV 0.96
![Page 23: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/23.jpg)
Radiographic Features Consistent with Anthrax
Input Data Variable
Transcribedchest radiographreport
Whether reportDescribes mediastinalfindings consistentwith anthrax
• Task: classify unseen chest radiograph reports as describing or not describing anthrax findings
• Gold standard: majority vote of 3 physicians
• Outcome measure: sensitivity, specificity, PPV, NPV
NLPApplication
![Page 24: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/24.jpg)
Mediastinal Evidence of Anthrax*
Revised IPS Model
Sens: 0.856
Spec: 0.988
PPV: 0.408
NPV: 0.999
Simple Keyword
Sens: 0.043
Spec: 0.999
PPV: 0.999
NPV: 0.979
*Chapman WW, Cooper GF, Hanbury P, Chapman BE, Harrison LH, Wagner MM. Creating A Text Classifier to Detect Radiology Reports Describing Mediastinal Findings Associated with Inhalational Anthrax and Other Disorders. J Am Med Inform Assoc 200310;494-503.
IPS Model
Sens: 0.351
Spec: 0.999
PPV: 0.965
NPV: 0.986
![Page 25: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/25.jpg)
Emergency Department Reports
![Page 26: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/26.jpg)
Respiratory Findings
• 71 findings from physician opinion and experience– Signs/Symptoms – dyspnea, cough, chest pain
– Physical findings – rales/crackles, chest dullness, fever
– Chest radiograph findings – pneumonia, pleural effusion
– Diseases – pneumonia, asthma
– Diseases that explain away respiratory findings – CHF, anxiety
• Detect findings with MetaMap* (NLM)• Test on 15 patient visits to ED (28 reports)
– Single physician as gold standard
*Aronson A. R. Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. Proc AMIA Symp. 2001:17-21.
![Page 27: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/27.jpg)
Detect Respiratory Findings with MetaMap*
MetaMap
Sens: 0.70
PPV: 0.55
Error Analysis– Domain lexicon – MetaMap mistake– Manual annotation– Contextual Discrimination
*Chapman WW, Fiszman M, Dowling JN, Chapman BE, Rindflesch TC. Identifying respiratory features fromEmergency departmnt reports for biosurveillance with MetaMap. Medinfo 2004 (in press).
![Page 28: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/28.jpg)
Summary: Technical Accuracy
• NLP techniques fairly sensitive and specific at extracting specific information from free-text– Chief complaints
• Extracting individual features
• Classifying complaints into categories
– Chest radiograph reports• Detecting pneumonia
• Detecting findings consistent with anthrax
– ED reports • Detecting fever
• More work is needed for generalizable solutions
![Page 29: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/29.jpg)
Diagnostic AccuracyCan we accurately diagnose patients from text?
NLPApplication
Variable values from NLP
ExpertSystem
Test Case Diagnoses from
Reference Standard
Test Case Diagnoses from
System
System Performance
Compare
Reference Standard
Variables from other
sources
Test Cases Text
![Page 30: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/30.jpg)
Chief Complaints
![Page 31: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/31.jpg)
Seven Syndromes from Chief Complaints
PositiveCases
Sensitivity Specificity PVP
Respiratory 34,916 0.63 0.94 0.44
Gastrointestinal 20,431 0.69 0.96 0.39
Neurological 7,393 0.68 0.93 0.12
Rash 2,232 0.47 0.99 0.22
Botulinic 1,961 0.30 0.99 0.14
Constitutional 10,603 0.46 0.97 0.22
Hemorrhagic 8,033 0.75 0.98 0.43
• Gold standard: ICD-9 primary discharge diagnoses• Test cases: 13 years of ED data
![Page 32: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/32.jpg)
Detecting Febrile Illness from Chief Complaints
Diagnostic Accuracy
Sensitivity: 0.61 (66/109)
Specificity: 1.0 (104/104)
Technical Accuracy for Fever from Chief Complaints: 100%
![Page 33: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/33.jpg)
Emergency Department Reports
![Page 34: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/34.jpg)
Detecting Febrile Illness from ED Reports*
• Keyword search– Fever synonyms– Temperature + value
• Accounts for negation with NegEx**http://omega.cbmi.upmc.edu/~chapman/NegEx.html
• Regular expression algorithm
• 6-word window from negation term
• Accounts for hypothetical findings– return, should, if, etc.
Sensitivity: 98% Specificity: 89%
* Chapman WW, Dowling JN, Wagner MM. Fever detection from free-text clinical records for biosurveillance.J Biomed Inform 2004;37(2):120-7.** Chapman WW, Bridewell W, Hanbury P, Cooper GF, Buchanan BG. A simple algorithm for identifyingNegated findings and diseases in discharge summaries. J Biomed Inform. 2001;34:301-10.
![Page 35: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/35.jpg)
Summary: Diagnostic Accuracy
• Good technical accuracy does not ensure good diagnostic accuracy– Depends on quality of input data
• The majority of syndromic patients can be detected from chief complaints
• Increased sensitivity requires more information– ED reports
• Case detection of one medical problem is doable– Fever
• Case detection for more complex syndromes requires more work– Pneumonic illness– SARS
![Page 36: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/36.jpg)
Outcome EfficacyCan we accurately detect outbreaks from text?
Requirements for Evaluation– Reference standard outbreak
– Textual data for patients involved in outbreak
Ivanov O, Gesteland P, Hogan W, Mundorff MB, Wagner MM. Detection of pediatric respiratory andGastrointestinal outbreaks from free-text chief complaints. Proc AMIA Annu Fall Symp 2003:318-22.
![Page 37: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/37.jpg)
Summary: Outcome Efficacy
• Very difficult to test
• Requires trust and cooperation
• Shown that chief complaints contain signal for outbreaks– Timelier that ICD-9 codes
![Page 38: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/38.jpg)
Are NLP Applications Good Enough for Biosurveillance?
1. How complex is the text?• Chief complaints easier than ED reports
2. What is the goal of the NLP technique?• Understand all temporal, anatomic, and diagnostic
relations of all clinical findings?• Unrealistic
• Extraction of a single variable or understanding of a limited set of variables?
• Realistic
3. Can the detection algorithms handle noise?• Small outbreaks require more accuracy in variables
• Inhalational Anthrax outbreak: 1 case = outbreak• Moderate to large outbreaks can handle noise
![Page 39: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/39.jpg)
Conclusions
![Page 40: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/40.jpg)
• Patient medical reports contain clinical data potentially relevant for outbreak detection– Free-text format
• Linguistic characteristics of patient medical reports must be considered to some extent
• Three types of evaluations necessary to understanding NLP’s contribution to biosurveillance– How well does NLP works in this domain?– How useful are different types of input data?
• Evaluation methods extensible to other domains to which NLP is applied
![Page 41: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/41.jpg)
Acknowledgments
• Mike Wagner• John Dowling• Oleg Ivanov • Bob Olszewski• Zhongwei Lu• Lee Christensen• Peter Haug
• Greg Cooper• Paul Hanbury• Rich Tsui• Jeremy Espino• Bill Hogan
![Page 42: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/42.jpg)
![Page 43: Natural Language Processing for Biosurveillance Wendy W. Chapman, PhD Center for Biomedical Informatics University of Pittsburgh.](https://reader030.fdocuments.in/reader030/viewer/2022032709/56649ebf5503460f94bca470/html5/thumbnails/43.jpg)
Outbreak Detection & Monitoring
Pre-hospital Syndromic Surveillance
•Over-the-counter medications•Absenteeism•Web queries•Call centers
Detect AbnormalPatterns in dataTo identify earliestStage of outbreak
Goa
lD
ata
Personinfected time
TraditionalCase detectionand reporting
DetectCases ofReportableDiseases
•Laboratory results
OutbreakMonitoring
DetermineActions toMinimizeEffect ofOutbreak
•CDC/WHO alerts•Surveillance system alerts•Newspaper articles•Email
= Free-text data source
Early Hospital Surveillance
•Chief complaints•ED Notes•Chest x-ray reports
Detect AbnormalPatterns in dataTo identify earlyStage of outbreak