A Quantitative and Qualitative Evaluation [-0.5mm]of ...
Transcript of A Quantitative and Qualitative Evaluation [-0.5mm]of ...
![Page 1: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/1.jpg)
A Quantitative and Qualitative Evaluationof Sentence Boundary Detection
for the Clinical Domain
Denis R Griffis, Chaitanya Shivade,Eric Fosler-Lussier, Albert M Lai
AMIA Joint Summits on Translational ScienceMarch 22, 2016
Department of Computer Science and Engineering
Department of Biomedical Informatics
![Page 2: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/2.jpg)
Outline
IntroductionChallenges in Sentence Boundary Detection (SBD)Motivation for Study
Evaluation
Discussion
Review
![Page 3: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/3.jpg)
What is Sentence Boundary Detection (SBD)?
UNIX SYSTEM LABS PICKS JUNE D-DAY, WOOSIBM, HP, DEC
Unix System Laboratories Inc has picked TuesdayJune 16 to launch Destiny, its desktop system nowofficially designated SVR4.2. A roll-out is expected onthe West Coast in either San Francisco or around SanJose, California, near the time of the XhibitionX-Windows show which will be held there that week.USL is hoping to collect an impressive array ofgodparents to stand witness. DEC, Hewlett-PackardCo and IBM have yet to agree to adopt the software,but USL is trying to get their representatives there ina show of solidarity and support for the operatingsystem. A magnanimous gesture from the founders ofthe Open Software Foundation is needed now to healany lingering breeches in the industry. Destiny is alsotheir one chance to beat back the forces of the Baronof Bellevue, Bill Gates, and his gathering MicrosoftNT hordes. Closed ranks would be USL’s pay-off forrecent concessions made to the Open SoftwareFoundation’s most important technologies.
→
UNIX SYSTEM LABS PICKS JUNE D-DAY, WOOSIBM, HP, DEC
Unix System Laboratories Inc has picked TuesdayJune 16 to launch Destiny, its desktop system nowofficially designated SVR4.2.
A roll-out is expected on the West Coast in eitherSan Francisco or around San Jose, California, nearthe time of the Xhibition X-Windows show whichwill be held there that week.
USL is hoping to collect an impressive array of god-parents to stand witness.
DEC, Hewlett-Packard Co and IBM have yet toagree to adopt the software, but USL is trying toget their representatives there in a show of solidarityand support for the operating system.
A magnanimous gesture from the founders of theOpen Software Foundation is needed now to healany lingering breeches in the industry.
Destiny is also their one chance to beat back theforces of the Baron of Bellevue, Bill Gates, and hisgathering Microsoft NT hordes.
Closed ranks would be USL’s pay-off for recent con-cessions made to the Open Software Foundation’smost important technologies.
![Page 4: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/4.jpg)
What is Sentence Boundary Detection (SBD)?
UNIX SYSTEM LABS PICKS JUNE D-DAY, WOOSIBM, HP, DEC
Unix System Laboratories Inc has picked TuesdayJune 16 to launch Destiny, its desktop system nowofficially designated SVR4.2. A roll-out is expected onthe West Coast in either San Francisco or around SanJose, California, near the time of the XhibitionX-Windows show which will be held there that week.USL is hoping to collect an impressive array ofgodparents to stand witness. DEC, Hewlett-PackardCo and IBM have yet to agree to adopt the software,but USL is trying to get their representatives there ina show of solidarity and support for the operatingsystem. A magnanimous gesture from the founders ofthe Open Software Foundation is needed now to healany lingering breeches in the industry. Destiny is alsotheir one chance to beat back the forces of the Baronof Bellevue, Bill Gates, and his gathering MicrosoftNT hordes. Closed ranks would be USL’s pay-off forrecent concessions made to the Open SoftwareFoundation’s most important technologies.
→
UNIX SYSTEM LABS PICKS JUNE D-DAY, WOOSIBM, HP, DEC
Unix System Laboratories Inc has picked TuesdayJune 16 to launch Destiny, its desktop system nowofficially designated SVR4.2.
A roll-out is expected on the West Coast in eitherSan Francisco or around San Jose, California, nearthe time of the Xhibition X-Windows show whichwill be held there that week.
USL is hoping to collect an impressive array of god-parents to stand witness.
DEC, Hewlett-Packard Co and IBM have yet toagree to adopt the software, but USL is trying toget their representatives there in a show of solidarityand support for the operating system.
A magnanimous gesture from the founders of theOpen Software Foundation is needed now to healany lingering breeches in the industry.
Destiny is also their one chance to beat back theforces of the Baron of Bellevue, Bill Gates, and hisgathering Microsoft NT hordes.
Closed ranks would be USL’s pay-off for recent con-cessions made to the Open Software Foundation’smost important technologies.
![Page 5: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/5.jpg)
SBD faces challenges in the clinical domain
6/10/1999 12:00:00 AMGASTROINTESTINAL BLEEDDISCHARGE DIAGNOSIS: SEPSIS.HISTORY OF THE PRESENT ILLNESS :She takes lisinopril / hydrochlorothiazide 20/25 mgp.o. q.d. , Vioxx 50 mg p.o. q.d. , Lipitor 10 mg p.o.q.d. , Nortriptyline 25 mg p.o. q.h.s. , Neurontin 300mg p.o. t.i.d. She had a regular heart rate andrhythm. Her gastrointestinal bleeding issues wereinvestigated with an upper endoscopy which revealedmultiple superficial gastic ulcerations consistent withan non-steroidal anti-inflammatory drugs gastopathy.Dictated By: MAULPLACKAGNELEEB, M.INACHELLE, M.D.
→
6/10/1999 12:00:00 AM
GASTROINTESTINAL BLEED
DISCHARGE DIAGNOSIS :
SEPSIS.
HISTORY OF THE PRESENT ILLNESS :
She takes lisinopril / hydrochlorothiazide 20/25 mgp.o. q.d. , Vioxx 50 mg p.o. q.d. , Lipitor 10 mgp.o. q.d. , Nortriptyline 25 mg p.o. q.h.s. , Neuron-tin 300 mg p.o. t.i.d.
She had a regular heart rate and rhythm.
Her gastrointestinal bleeding issues were investigatedwith an upper endoscopy which revealed multiplesuperficial gastic ulcerations consistent with an non-steroidal anti-inflammatory drugs gastopathy.
Dictated By :
MAULPLACKAGNELEEB, M. INACHELLE, M.D.
![Page 6: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/6.jpg)
SBD faces challenges in the clinical domain
6/10/1999 12:00:00 AMGASTROINTESTINAL BLEEDDISCHARGE DIAGNOSIS: SEPSIS.HISTORY OF THE PRESENT ILLNESS :She takes lisinopril / hydrochlorothiazide 20/25 mgp.o. q.d. , Vioxx 50 mg p.o. q.d. , Lipitor 10 mg p.o.q.d. , Nortriptyline 25 mg p.o. q.h.s. , Neurontin 300mg p.o. t.i.d. She had a regular heart rate andrhythm. Her gastrointestinal bleeding issues wereinvestigated with an upper endoscopy which revealedmultiple superficial gastic ulcerations consistent withan non-steroidal anti-inflammatory drugs gastopathy.Dictated By: MAULPLACKAGNELEEB, M.INACHELLE, M.D.
→
6/10/1999 12:00:00 AM
GASTROINTESTINAL BLEED
DISCHARGE DIAGNOSIS :
SEPSIS.
HISTORY OF THE PRESENT ILLNESS :
She takes lisinopril / hydrochlorothiazide 20/25 mgp.o. q.d. , Vioxx 50 mg p.o. q.d. , Lipitor 10 mgp.o. q.d. , Nortriptyline 25 mg p.o. q.h.s. , Neuron-tin 300 mg p.o. t.i.d.
She had a regular heart rate and rhythm.
Her gastrointestinal bleeding issues were investigatedwith an upper endoscopy which revealed multiplesuperficial gastic ulcerations consistent with an non-steroidal anti-inflammatory drugs gastopathy.
Dictated By :
MAULPLACKAGNELEEB, M. INACHELLE, M.D.
![Page 7: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/7.jpg)
Example “sentences” from different domains
NewswireUSL has had Destiny, initially con-ceived for Intel Corp platforms, inbeta test for some weeks and shouldstart regular deliveries to its OEMcustomers in July.
Biomedical abstractsThe 5’ sequences up to nucleotide -120 of the human and murine IL-16genes share >84% sequence homol-ogy and harbor promoter elementsfor constitutive and inducible tran-scription in T cells.
Speech (telephone)Yeah. Uh-huh. W-, uh, the, the callwas probably for her.
Clinical textThe hCG on admission was 30,710and on 1/19 was 805.
Note: the term “sentence” doesn’t always make sense.
Different domains prefer different kinds of segmentation.
![Page 8: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/8.jpg)
Example “sentences” from different domains
NewswireUSL has had Destiny, initially con-ceived for Intel Corp platforms, inbeta test for some weeks and shouldstart regular deliveries to its OEMcustomers in July.
Biomedical abstractsThe 5’ sequences up to nucleotide -120 of the human and murine IL-16genes share >84% sequence homol-ogy and harbor promoter elementsfor constitutive and inducible tran-scription in T cells.
Speech (telephone)Yeah. Uh-huh. W-, uh, the, the callwas probably for her.
Clinical textThe hCG on admission was 30,710and on 1/19 was 805.
Note: the term “sentence” doesn’t always make sense.
Different domains prefer different kinds of segmentation.
![Page 9: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/9.jpg)
SBD needs to adapt to different assumptions
Different text domains have different expectations of
I structure (long/short sentences, discrete sections)
I formatting (variable case, unusual numeric patterns)
GENIAThe 5’ sequences up to nucleotide -120 of the human and murine IL-16 genes share>84% sequence homology and harbor promoter elements for constitutive andinducible transcription in T cells.
i2b2ALT (SGPT) - 249 AST (SGOT) - 147 LD (LDH) - 241 ALK PHOS - 230 AMYLASE- 28 TOT BILI - 0.9 LIPASE - 12 ALBUMIN - 2.6
There is no one-size-fits-all approach!
![Page 10: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/10.jpg)
SBD needs to adapt to different assumptions
Different text domains have different expectations of
I structure (long/short sentences, discrete sections)
I formatting (variable case, unusual numeric patterns)
GENIAThe 5’ sequences up to nucleotide -120 of the human and murine IL-16 genes share>84% sequence homology and harbor promoter elements for constitutive andinducible transcription in T cells.
i2b2ALT (SGPT) - 249 AST (SGOT) - 147 LD (LDH) - 241 ALK PHOS - 230 AMYLASE- 28 TOT BILI - 0.9 LIPASE - 12 ALBUMIN - 2.6
There is no one-size-fits-all approach!
![Page 11: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/11.jpg)
SBD errors have impact far downstream
SBD
POS tagging
Dependency tagging
Named Entity Recognition
Medication Extraction
Clinical Trial Eligibility
Lisinopril./ Hydrochlorothiazide 10 mg., po t.i.d.
Sentence 1 Sentence 1Sentence 2Sentence 3NNPNNCD NN NNNNX
Missing
C0065374
X
C0020261
X
C0717824
Amount: 10 mg
Drug: Lisinopril/Hydrochlorothiazide
Method: po
Frequency: t.i.d
Drug
Amount
MtdFrq
Inclusion Criteria. . .Patient on Lisinopril/Hydrochlorothiazide at
hospital discharge.. . .
![Page 12: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/12.jpg)
SBD errors have impact far downstream
SBD
POS tagging
Dependency tagging
Named Entity Recognition
Medication Extraction
Clinical Trial Eligibility
Lisinopril./ Hydrochlorothiazide 10 mg., po t.i.d.
Sentence 1
Sentence 1Sentence 2Sentence 3NNPNNCD NN NNNNX
Missing
C0065374
X
C0020261
X
C0717824
Amount: 10 mg
Drug: Lisinopril/Hydrochlorothiazide
Method: po
Frequency: t.i.d
Drug
Amount
MtdFrq
Inclusion Criteria. . .Patient on Lisinopril/Hydrochlorothiazide at
hospital discharge.. . .
![Page 13: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/13.jpg)
SBD errors have impact far downstream
SBD
POS tagging
Dependency tagging
Named Entity Recognition
Medication Extraction
Clinical Trial Eligibility
Lisinopril. / Hydrochlorothiazide 10 mg. , po t.i.d
Sentence 1
Sentence 1 Sentence 2 Sentence 3
NNP NN CD NN NNNNX
Missing
C0065374
X
C0020261
X
C0717824
Amount: 10 mg
Drug: Lisinopril/Hydrochlorothiazide
Method: po
Frequency: t.i.d
Drug
Amount
Mtd Frq
Inclusion Criteria. . .Patient on Lisinopril/Hydrochlorothiazide at
hospital discharge.. . .
![Page 14: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/14.jpg)
SBD errors have impact far downstream
SBD
POS tagging
Dependency tagging
Named Entity Recognition
Medication Extraction
Clinical Trial Eligibility
Lisinopril. / Hydrochlorothiazide 10 mg. , po t.i.d
Sentence 1Sentence 1 Sentence 2 Sentence 3
NNP NN CD NN NNNN
X
Missing
C0065374
X
C0020261
X
C0717824
Amount: 10 mg
Drug: Lisinopril/Hydrochlorothiazide
Method: po
Frequency: t.i.d
Drug
Amount
Mtd Frq
Inclusion Criteria. . .Patient on Lisinopril/Hydrochlorothiazide at
hospital discharge.. . .
![Page 15: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/15.jpg)
SBD errors have impact far downstream
SBD
POS tagging
Dependency tagging
Named Entity Recognition
Medication Extraction
Clinical Trial Eligibility
Lisinopril. / Hydrochlorothiazide 10 mg. , po t.i.d
Sentence 1Sentence 1 Sentence 2 Sentence 3NNP NN CD NN NNNN
X
Missing
C0065374
X
C0020261
X
C0717824
Amount: 10 mg
Drug: Lisinopril/Hydrochlorothiazide
Method: po
Frequency: t.i.d
Drug
Amount
Mtd Frq
Inclusion Criteria. . .Patient on Lisinopril/Hydrochlorothiazide at
hospital discharge.. . .
![Page 16: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/16.jpg)
SBD errors have impact far downstream
SBD
POS tagging
Dependency tagging
Named Entity Recognition
Medication Extraction
Clinical Trial Eligibility
Lisinopril. / Hydrochlorothiazide 10 mg. , po t.i.d
Sentence 1Sentence 1 Sentence 2 Sentence 3NNP NN CD NN NNNNX
Missing
C0065374
X
C0020261
X
C0717824
Amount: 10 mg
Drug: Lisinopril/Hydrochlorothiazide
Method: po
Frequency: t.i.d
Drug
Amount
Mtd Frq
Inclusion Criteria. . .Patient on Lisinopril/Hydrochlorothiazide at
hospital discharge.. . .
![Page 17: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/17.jpg)
SBD errors have impact far downstream
SBD
POS tagging
Dependency tagging
Named Entity Recognition
Medication Extraction
Clinical Trial Eligibility
Lisinopril. / Hydrochlorothiazide 10 mg. , po t.i.d
Sentence 1Sentence 1 Sentence 2 Sentence 3NNP NN CD NN NNNNX
Missing
C0065374
X
C0020261
X
C0717824
Amount: 10 mg
Drug: Lisinopril/Hydrochlorothiazide
Method: po
Frequency: t.i.d
Drug
Amount
Mtd Frq
Inclusion Criteria. . .Patient on Lisinopril/Hydrochlorothiazide at
hospital discharge.. . .
![Page 18: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/18.jpg)
SBD errors have impact far downstream
SBD
POS tagging
Dependency tagging
Named Entity Recognition
Medication Extraction
Clinical Trial Eligibility
Lisinopril. / Hydrochlorothiazide 10 mg. , po t.i.d
Sentence 1Sentence 1 Sentence 2 Sentence 3NNP NN CD NN NNNNX
Missing
C0065374
X
C0020261
X
C0717824
Amount: 10 mg
Drug: Lisinopril/Hydrochlorothiazide
Method: po
Frequency: t.i.d
Drug
Amount
Mtd Frq
Inclusion Criteria. . .Patient on Lisinopril/Hydrochlorothiazide at
hospital discharge.. . .
![Page 19: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/19.jpg)
Why is it time to re-evaluate SBD?
SBD is a critical first step for many NLP tasks.
SBD is often treated as “solved” and done with off-the-shelf toolkits.
But this can lead to serious errors!
Our goal:Evaluate off-the-shelf toolkits on SBD,
focusing on clinical text.
![Page 20: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/20.jpg)
Why is it time to re-evaluate SBD?
SBD is a critical first step for many NLP tasks.
SBD is often treated as “solved” and done with off-the-shelf toolkits.
But this can lead to serious errors!
Our goal:Evaluate off-the-shelf toolkits on SBD,
focusing on clinical text.
![Page 21: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/21.jpg)
Why is it time to re-evaluate SBD?
SBD is a critical first step for many NLP tasks.
SBD is often treated as “solved” and done with off-the-shelf toolkits.
But this can lead to serious errors!
Our goal:Evaluate off-the-shelf toolkits on SBD,
focusing on clinical text.
![Page 22: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/22.jpg)
Why is it time to re-evaluate SBD?
SBD is a critical first step for many NLP tasks.
SBD is often treated as “solved” and done with off-the-shelf toolkits.
But this can lead to serious errors!
Our goal:Evaluate off-the-shelf toolkits on SBD,
focusing on clinical text.
![Page 23: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/23.jpg)
Why is it time to re-evaluate SBD?
SBD is a critical first step for many NLP tasks.
SBD is often treated as “solved” and done with off-the-shelf toolkits.
But this can lead to serious errors!
Our goal:Evaluate off-the-shelf toolkits on SBD,
focusing on clinical text.
![Page 24: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/24.jpg)
Outline
Introduction
EvaluationThe toolkitsThe datasetsEvaluation method
Discussion
Review
![Page 25: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/25.jpg)
The toolkits
Toolkit Training CorporaStanford CoreNLP PTB1, GENIA2, other Stanford corpora
Lingpipe MEDLINE abstracts, general text
Splitta PTB
SPECIALIST SPECIALIST lexicon3
cTAKES GENIA, PTB, Mayo Clinic EMR
1Penn Treebank (PTB): corpus of Wall Street Journal articles2GENIA: corpus of biomedical abstracts3SPECIALIST lexicon: vocabulary from biomedical and general English
General-domain corpora Biomedical corpora Clinical text corpora
![Page 26: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/26.jpg)
The datasets
Well-formedtext corpora
Non-standardtext corpora
General-domain BNC Switchboard
Biomedical GENIA i2b2
BNC Mixed-domain British English
Switchboard Spoken English telephone transcripts
GENIA Biomedical abstracts
i2b2 Clinical EHR notes
![Page 27: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/27.jpg)
How we evaluated the toolkits
1. Run each toolkit on each corpus
2. Extract predicted sentence bounds from the output
3. Compare beginning and ending of each sentence against goldstandard
Example text
[
Patient exhibits mild symptoms.
][
m.g. of aspirin administered.
]
Gold standard PredictedBgn End Bgn End
10 40 10 4041 75 41 43- - 44 75
True Positives: 4False Positives: 2
![Page 28: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/28.jpg)
How we evaluated the toolkits
1. Run each toolkit on each corpus
2. Extract predicted sentence bounds from the output
3. Compare beginning and ending of each sentence against goldstandard
Example text
[
Patient exhibits mild symptoms.
][
m.g. of aspirin administered.
]
Gold standard PredictedBgn End Bgn End
10 40 10 4041 75 41 43- - 44 75
True Positives: 4False Positives: 2
![Page 29: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/29.jpg)
How we evaluated the toolkits
1. Run each toolkit on each corpus
2. Extract predicted sentence bounds from the output
3. Compare beginning and ending of each sentence against goldstandard
Example text
[
Patient exhibits mild symptoms.
][
m.g. of aspirin administered.
]
Gold standard PredictedBgn End Bgn End
10 40 10 4041 75 41 43- - 44 75
True Positives: 4False Positives: 2
![Page 30: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/30.jpg)
How we evaluated the toolkits
1. Run each toolkit on each corpus
2. Extract predicted sentence bounds from the output
3. Compare beginning and ending of each sentence against goldstandard
Example text
[
Patient exhibits mild symptoms.
][
12.3* m.g. of aspirinadministered.
]
Gold standard PredictedBgn End Bgn End
10 40 10 4041 75 41 43- - 44 75
True Positives: 4False Positives: 2
![Page 31: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/31.jpg)
How we evaluated the toolkits
1. Run each toolkit on each corpus
2. Extract predicted sentence bounds from the output
3. Compare beginning and ending of each sentence against goldstandard
Example text
[ Patient exhibits mild symptoms. ][ 12.3* m.g. of aspirinadministered. ]
Gold standard PredictedBgn End Bgn End10 40
10 40
41 75
41 43- - 44 75
True Positives: 4False Positives: 2
![Page 32: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/32.jpg)
How we evaluated the toolkits
1. Run each toolkit on each corpus
2. Extract predicted sentence bounds from the output
3. Compare beginning and ending of each sentence against goldstandard
Example text
[ Patient exhibits mild symptoms. ][ 12.] [3* m.g. of aspirinadministered. ]
Gold standard PredictedBgn End Bgn End10 40 10 4041 75 41 43- - 44 75
True Positives: 4False Positives: 2
![Page 33: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/33.jpg)
F1-score of each toolkit on each corpus
Well-formed Non-standardToolkit BNC GENIA SWB i2b2
Stanford 0.82 0.98 0.45 0.43LingpipeGeneral 0.73 0.96 0.42 0.42LingpipeMedline 0.72 0.99 0.43 0.41SplittaSVM - 0.97 0.39 0.43SplittaNaiveBayes - 0.99 0.39 0.43SPECIALIST 0.74 0.92 0.46 0.56cTAKES 0.74 0.68 0.55 0.95
What’s going on here?
![Page 34: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/34.jpg)
F1-score of each toolkit on each corpus
Well-formed Non-standardToolkit BNC GENIA SWB i2b2
Stanford 0.82 0.98 0.45 0.43LingpipeGeneral 0.73 0.96 0.42 0.42LingpipeMedline 0.72 0.99 0.43 0.41SplittaSVM - 0.97 0.39 0.43SplittaNaiveBayes - 0.99 0.39 0.43SPECIALIST 0.74 0.92 0.46 0.56cTAKES 0.74 0.68 0.55 0.95
What’s going on here?
![Page 35: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/35.jpg)
F1-score of each toolkit on each corpus
Well-formed Non-standardToolkit BNC GENIA SWB i2b2
Stanford 0.82 0.98 0.45 0.43LingpipeGeneral 0.73 0.96 0.42 0.42LingpipeMedline 0.72 0.99 0.43 0.41SplittaSVM - 0.97 0.39 0.43SplittaNaiveBayes - 0.99 0.39 0.43SPECIALIST 0.74 0.92 0.46 0.56cTAKES 0.74 0.68 0.55 0.95
What’s going on here?
![Page 36: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/36.jpg)
F1-score of each toolkit on each corpus
Well-formed Non-standardToolkit BNC GENIA SWB i2b2
Stanford 0.82 0.98 0.45 0.43LingpipeGeneral 0.73 0.96 0.42 0.42LingpipeMedline 0.72 0.99 0.43 0.41SplittaSVM - 0.97 0.39 0.43SplittaNaiveBayes - 0.99 0.39 0.43SPECIALIST 0.74 0.92 0.46 0.56cTAKES 0.74 0.68 0.55 0.95
What’s going on here?
![Page 37: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/37.jpg)
F1-score of each toolkit on each corpus
Well-formed Non-standardToolkit BNC GENIA SWB i2b2
Stanford 0.82 0.98 0.45 0.43LingpipeGeneral 0.73 0.96 0.42 0.42LingpipeMedline 0.72 0.99 0.43 0.41SplittaSVM - 0.97 0.39 0.43SplittaNaiveBayes - 0.99 0.39 0.43SPECIALIST 0.74 0.92 0.46 0.56cTAKES 0.74 0.68 0.55 0.95
What’s going on here?
![Page 38: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/38.jpg)
F1-score of each toolkit on each corpus
Well-formed Non-standardToolkit BNC GENIA SWB i2b2
Stanford 0.82 0.98 0.45 0.43LingpipeGeneral 0.73 0.96 0.42 0.42LingpipeMedline 0.72 0.99 0.43 0.41SplittaSVM - 0.97 0.39 0.43SplittaNaiveBayes - 0.99 0.39 0.43SPECIALIST 0.74 0.92 0.46 0.56cTAKES 0.74 0.68 0.55 0.95
What’s going on here?
![Page 39: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/39.jpg)
Outline
Introduction
Evaluation
Discussion
Review
![Page 40: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/40.jpg)
SBD is sensitive to punctuation usage
![Page 41: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/41.jpg)
SBD is sensitive to punctuation usage
![Page 42: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/42.jpg)
Sentence length matters
Signed by: DR. Robert Downey on: (WED 2016-05-18 5:18PM)
Sentence 1
cTAKES
Signed by: DR. Robert Downey on: (WED 2016-05-18 5:18PM)
Stanford
Signed by: DR. Robert Downey on: (WED 2016-05-18 05:18PM) . . .
![Page 43: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/43.jpg)
Sentence length matters
Signed by: DR. Robert Downey on: (WED 2016-05-18 5:18PM)
Sentence 1
cTAKES
Signed by: DR. Robert Downey on: (WED 2016-05-18 5:18PM)
Stanford
Signed by: DR. Robert Downey on: (WED 2016-05-18 05:18PM) . . .
![Page 44: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/44.jpg)
Sentence length matters
Signed by: DR. Robert Downey on: (WED 2016-05-18 5:18PM)
Sentence 1
cTAKES
Signed by: DR. Robert Downey on: (WED 2016-05-18 5:18PM)
Stanford
Signed by: DR. Robert Downey on: (WED 2016-05-18 05:18PM) . . .
![Page 45: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/45.jpg)
Abbreviations are still problematic
Unfamiliar variants“t.i.d” is fine, “t.i.d.” considered sentence terminal.
New abbreviations and initialsSt. CyresJoseph R. Cowdon
![Page 46: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/46.jpg)
Abbreviations are still problematic
Unfamiliar variants“t.i.d” is fine, “t.i.d.” considered sentence terminal.
New abbreviations and initialsSt. CyresJoseph R. Cowdon
![Page 47: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/47.jpg)
Abbreviations are still problematic
Unfamiliar variants“t.i.d” is fine, “t.i.d.” considered sentence terminal.
New abbreviations and initialsSt. CyresJoseph R. Cowdon
![Page 48: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/48.jpg)
Non-standard formatting causes errors
Case errors. . . by DR. Melvin N.I. LICHTENBERGER. . .
. . . human nm23-H2 gene product. nm23 gene. . .
→ Extra breaks
→ Missed break
Numeric formatting (e.g. readings)
. . . 1.1 +/- 0.4 10(-18) . . .
. . . 11.4* . . .cause false positives
Extra headers. . . murine erythroleukemia (MEL) cells . . .
(i) item 1
(ii) item 2
→ Caps in parens
→ Lists
![Page 49: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/49.jpg)
Non-standard formatting causes errors
Case errors. . . by DR. Melvin N.I. LICHTENBERGER. . .
. . . human nm23-H2 gene product. nm23 gene. . .
→ Extra breaks
→ Missed break
Numeric formatting (e.g. readings)
. . . 1.1 +/- 0.4 10(-18) . . .
. . . 11.4* . . .cause false positives
Extra headers. . . murine erythroleukemia (MEL) cells . . .
(i) item 1
(ii) item 2
→ Caps in parens
→ Lists
![Page 50: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/50.jpg)
Non-standard formatting causes errors
Case errors. . . by DR. Melvin N.I. LICHTENBERGER. . .
. . . human nm23-H2 gene product. nm23 gene. . .
→ Extra breaks
→ Missed break
Numeric formatting (e.g. readings)
. . . 1.1 +/- 0.4 10(-18) . . .
. . . 11.4* . . .cause false positives
Extra headers. . . murine erythroleukemia (MEL) cells . . .
(i) item 1
(ii) item 2
→ Caps in parens
→ Lists
![Page 51: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/51.jpg)
Non-standard formatting causes errors
Case errors. . . by DR. Melvin N.I. LICHTENBERGER. . .
. . . human nm23-H2 gene product. nm23 gene. . .
→ Extra breaks
→ Missed break
Numeric formatting (e.g. readings)
. . . 1.1 +/- 0.4 10(-18) . . .
. . . 11.4* . . .cause false positives
Extra headers. . . murine erythroleukemia (MEL) cells . . .
(i) item 1
(ii) item 2
→ Caps in parens
→ Lists
![Page 52: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/52.jpg)
Outline
Introduction
Evaluation
Discussion
Review
![Page 53: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/53.jpg)
Recap: evaluation of SBD toolkits
Our goal
Evaluate off-the-shelf toolkits on SBDwith a focus towards clinical text.
To that end
We ran severalpopular tools on avariety of datasets
We found domainsensitivity and pooroverall performance
on clinical text.
![Page 54: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/54.jpg)
Recap: evaluation of SBD toolkits
Our goal
Evaluate off-the-shelf toolkits on SBDwith a focus towards clinical text.
To that end
We ran severalpopular tools on avariety of datasets
We found domainsensitivity and pooroverall performance
on clinical text.
![Page 55: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/55.jpg)
Recap: evaluation of SBD toolkits
Our goal
Evaluate off-the-shelf toolkits on SBDwith a focus towards clinical text.
To that end
We ran severalpopular tools on avariety of datasets
We found domainsensitivity and pooroverall performance
on clinical text.
![Page 56: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/56.jpg)
Main takeaways
SBD in clinical text faces the challenges of:
I Different patterns of punctuation usage
I Different structure and length of “sentences”
I Different assumptions about text formatting
I Different definition of a useful “sentence”
SBD errors negatively impact downstreambiomedical applications, e.g.
I Medical event recognition
I Drug interaction mining
I Automated clinical trialeligibility screening
I etc.
![Page 57: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/57.jpg)
Main takeaways
SBD in clinical text faces the challenges of:
I Different patterns of punctuation usage
I Different structure and length of “sentences”
I Different assumptions about text formatting
I Different definition of a useful “sentence”
SBD errors negatively impact downstreambiomedical applications, e.g.
I Medical event recognition
I Drug interaction mining
I Automated clinical trialeligibility screening
I etc.
![Page 58: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/58.jpg)
Main takeaways
SBD in clinical text faces the challenges of:
I Different patterns of punctuation usage
I Different structure and length of “sentences”
I Different assumptions about text formatting
I Different definition of a useful “sentence”
SBD errors negatively impact downstreambiomedical applications, e.g.
I Medical event recognition
I Drug interaction mining
I Automated clinical trialeligibility screening
I etc.
![Page 59: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/59.jpg)
Main takeaways
SBD in clinical text faces the challenges of:
I Different patterns of punctuation usage
I Different structure and length of “sentences”
I Different assumptions about text formatting
I Different definition of a useful “sentence”
SBD errors negatively impact downstreambiomedical applications, e.g.
I Medical event recognition
I Drug interaction mining
I Automated clinical trialeligibility screening
I etc.
![Page 60: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/60.jpg)
Main takeaways
SBD in clinical text faces the challenges of:
I Different patterns of punctuation usage
I Different structure and length of “sentences”
I Different assumptions about text formatting
I Different definition of a useful “sentence”
SBD errors negatively impact downstreambiomedical applications, e.g.
I Medical event recognition
I Drug interaction mining
I Automated clinical trialeligibility screening
I etc.
![Page 61: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/61.jpg)
Main takeaways
SBD in clinical text faces the challenges of:
I Different patterns of punctuation usage
I Different structure and length of “sentences”
I Different assumptions about text formatting
I Different definition of a useful “sentence”
SBD errors negatively impact downstreambiomedical applications, e.g.
I Medical event recognition
I Drug interaction mining
I Automated clinical trialeligibility screening
I etc.
![Page 62: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/62.jpg)
Where do we go from here?
Long-term
I Work on developing lightweight SBD methods that can beeasily adapted to new domains.
I Create and share datasets of clinical text annotated for SBD.
Short-term
I Add a training pipeline to cTAKES and allow for customabbreviation lists.
I Explore new rules for Stanford CoreNLP to work well onclinical text.
![Page 63: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/63.jpg)
Where do we go from here?
Long-term
I Work on developing lightweight SBD methods that can beeasily adapted to new domains.
I Create and share datasets of clinical text annotated for SBD.
Short-term
I Add a training pipeline to cTAKES and allow for customabbreviation lists.
I Explore new rules for Stanford CoreNLP to work well onclinical text.
![Page 64: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/64.jpg)
Where do we go from here?
Long-term
I Work on developing lightweight SBD methods that can beeasily adapted to new domains.
I Create and share datasets of clinical text annotated for SBD.
Short-term
I Add a training pipeline to cTAKES and allow for customabbreviation lists.
I Explore new rules for Stanford CoreNLP to work well onclinical text.
![Page 65: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/65.jpg)
Where do we go from here?
Long-term
I Work on developing lightweight SBD methods that can beeasily adapted to new domains.
I Create and share datasets of clinical text annotated for SBD.
Short-term
I Add a training pipeline to cTAKES and allow for customabbreviation lists.
I Explore new rules for Stanford CoreNLP to work well onclinical text.
![Page 66: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/66.jpg)
Where do we go from here?
Long-term
I Work on developing lightweight SBD methods that can beeasily adapted to new domains.
I Create and share datasets of clinical text annotated for SBD.
Short-term
I Add a training pipeline to cTAKES and allow for customabbreviation lists.
I Explore new rules for Stanford CoreNLP to work well onclinical text.
![Page 67: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/67.jpg)
Where do we go from here?
Long-term
I Work on developing lightweight SBD methods that can beeasily adapted to new domains.
I Create and share datasets of clinical text annotated for SBD.
Short-term
I Add a training pipeline to cTAKES and allow for customabbreviation lists.
I Explore new rules for Stanford CoreNLP to work well onclinical text.
![Page 68: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/68.jpg)
Where do we go from here?
Long-term
I Work on developing lightweight SBD methods that can beeasily adapted to new domains.
I Create and share datasets of clinical text annotated for SBD.
Short-term
I Add a training pipeline to cTAKES and allow for customabbreviation lists.
I Explore new rules for Stanford CoreNLP to work well onclinical text.
![Page 69: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/69.jpg)
Acknowledgments
Co-authors:
Chaitanya Shivade
Eric Fosler-Lussier*
Albert M Lai*
*Co-advisors
Supported by:
I Award Number Grant R01LM011116from the National Library ofMedicine
I The Intramural Research Program ofthe National Institutes of Health,Clinical Research Center
I Inter-Agency Agreement with the USSocial Security Administration
Source code available at:http://github.com/drgriffis/sbd-evaluation
Department of Computer Science and Engineering
Department of Biomedical Informatics
![Page 70: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/70.jpg)
Thank you!
Source code available at:http://github.com/drgriffis/sbd-evaluation
Contact Info:
Denis Griffis | [email protected]
Department of Computer Science and Engineering
Department of Biomedical Informatics
![Page 71: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/71.jpg)
Supplemental: Runtime of each toolkit on i2b2 corpus
![Page 72: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/72.jpg)
Supplemental: Corpus details
Corpus # of Documents # of Sentences Avg. # tokens
BNC 4,049 6,027,378 16.1
Switchboard 650 110,504 7.4
GENIA 1,999 16,479 24.4
i2b2 426 43,940 9.5
![Page 73: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/73.jpg)
Supplemental: Detailed errors
![Page 74: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/74.jpg)
Supplemental: Detailed errors
![Page 75: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/75.jpg)
Supplemental: Detailed errors
![Page 76: A Quantitative and Qualitative Evaluation [-0.5mm]of ...](https://reader030.fdocuments.in/reader030/viewer/2022012704/61a65bdd2fda62139a171962/html5/thumbnails/76.jpg)
Supplemental: Detailed errors