Eiji Aramaki (Kyoto university) Mizuki Morita (The university of Tokyo) Tomoko Ohkuma (Fuji Xerox)...

Eiji Aramaki (Kyoto university) Mizuki Morita (The university of Tokyo)Tomoko Ohkuma (Fuji Xerox)Yoshinobu Kano (Shizuoka university)

Overview of MedNLP-2

Why are we dealing with Medical records?Medical records contain rich clinical information as in text form

Medical records contain rich clinical information as textBUT: The amount is more than one researcher can handle → requires ICT (NLP)

Our goal is to develop the fundamental techniques for NLP in the medical field.ALSO, we are aiming to develop methodology, and publish the standard tools for the medical NLP.

Who are the organizers?

NLP Researcher NLP Researcher(Fuji Xerox)

Bioinformatics

• Organizers cover both academic researchers and a company member

• Covering various fields (not only pure NLP/IR but also bioinformatics & framework making),

FrameworkTool sharing

Bioinformatics

Company viewpoint

NLP Researcher

Medical NLP

Who are the organizers?

NLP Researcher NLP Researcher(Fuji Xerox)

Bioinformatics

• Because MedNLP targets on two aspects: computer science & medical application, this them is suitable for such multiple aspects

FrameworkTool sharing

Bioinformatics

Company viewpoint

NLP Researcher

Medical NLP

Overview

• Background• Material• Task Design• Overview of Task 1• Overview of Task 2• What’ the Next

One & Only Non-English Medical Shared Task

• Medical Shared Task– Image CLEFmed (2005-) Image– I2b2 NLP (2006) English– TREC Medical Records Track (2011) English– CLEFeHealth (2013) English– MedNLP (2011) Japanese (& non-English) :

• Why are not many non-English medical tasks available?

Medical Record contains Privacy Information

• In US, HIPPA clearly defines what is privacy, consisting of 18 items (name, telephone number, e-mail address, face picture….)– Once privacy information is removed, it

can be used freely→ SO: many English Health Records can be available

• In contrast, Japan is still conservative• we do not have such clear privacy

guideline for medical text• This becomes a heavy barrier for

research use for medical text

To break the barrier:2 types of dummy medical records

• (1) Dummy (virtual) Records– We asked volunteers, who are MDs, to write

records assuming dummy (virtual) patients– Then, we bought the records

• (2) Exam Texts– Question texts of the National Medical Exam

(=“ 医師国家試験” ) for doctors.

Exam basically consists of multiple question like SAT test in US or “center shiken” in JapanMost of question is give in the form of short sentence. BUT…

Part A .Medical Examination for Doctors (2005)

Some of question contains rich information on a patient, which is called “case based question”That style is very similar to clinical recordSo, we convert the data to corpus

Exam Texts

Conversion process is 2 folds: Question style expression, such as multiple options, are removedWs also add Named Entity, date time, to the corpus

Quantity & Quality of Dummy dataMedNLP-1 MedNLP-2

Disorder of the Alimentary Tract

4 19

Liver, Biliary Tract & Pancreas

2 12

Cardiovascular System 12 23Endocrinology, Metabolism & Nutrition

5 17

Disorders of the Kidney & Urinary Tract

4 14

Immune System & Immune-Mediated Injury

5 17

Disorders of the Hematopoietic System

1 13

Infectious Disease 6 15Disorders of the Respiratory System

11 26

TOTAL 50 156

While MedNLP-1 does not covers several clinical domains enough, MedNLP-2 covers all domains

It was hard to distinguish even forMDs

Accuracy

Medical (physician) (n=2)

60.0%

Non medical (n=3) 56.3%

10 dummy records 10 real records

To validate the quality, we ask MDs to classify the dummy records from the mixed corpus

Task Design

De identification

NER

京大病院来院 5 日前から腹が痛むとのこと

MilestoneM

ed

NLP

-2Coding

Decision Support

■■大病院来院 5 日前から腹が痛むとのこと

■■大病院来院 5 日前から腹痛とのこと

■■大病院来院 5 日前から R104 とのこと

What kind of Task is required?M

ed

NLP

-1

MedNLP-2 targets on The 2nd step & 3rd step.

Output Example

MedNLP-1 MedNLP-2

Given a raw text

Participants

Participants increased!Task MedNLP-1 MedNLP-2

De-identification 6 groups (11 systems)

-

NER 11 groups(15 systems)

10 groups (24 systems)

ICD-coding - 9 groups(19 systems)

Free 1 groups(1 systems)

2 groups(2 systems)

The number of groups is the same to the previous MedNLP-1

The number of systems increased much

• Surprisingly, In total, MedNLP-2 had 12 groups and 45 systems!• One of the most active tasks in NTCIR• More Surprisingly: ICD-coding task, which is a medical specific

task, also almost 20 submissions.• This indicates that NLP people pay much attention to find the

way to reach the medical application.

Lists of MedNLP-2 Participants北陸先端科学技術大学院大学JAIST

北海道大学Hokkaido University

京都大学Kyoto University

岡山大学Okayama Prefectural University

東京大学The University of Tokyo

奈良先端科学技術大学院大学Nara Institute of Science and Technology

安田女子大学Yasuda Women's College

国立中央大学（台湾）National Central University

朝陽科技大学（台湾）Chaoyang University of Technology

南京大学（中国）Nanjing University

中央研究院（台湾）Academia Sinica

ダブリン大学（英国）Dublin City University

日本ユニシスNihon Unisys, Ltd

日立中央研究所Hitachi, Ltd.

NTT 研究所NTT Science and Core Technology Laboratory Group

海外Oversea

企業Company

大学Academic

Participants have various background just as the organizers

Lists of MedNLP-2 Participants北陸先端科学技術大学院大学JAIST

北海道大学Hokkaido University

京都大学Kyoto University

岡山大学Okayama Prefectural University

東京大学The University of Tokyo

奈良先端科学技術大学院大学Nara Institute of Science and Technology

安田女子大学Yasuda Women's College

国立中央大学（台湾）National Central University

朝陽科技大学（台湾）Chaoyang University of Technology

南京大学（中国）Nanjing University

中央研究院（台湾）Academia Sinica

ダブリン大学（英国）Dublin City University

日本ユニシスNihon Unisys, Ltd

日立中央研究所Hitachi, Ltd.

NTT 研究所NTT Science and Core Technology Laboratory Group

海外Oversea

企業Company

大学Academic

We are very happy to have five submission from oversea→ Although the material is Japanese language only, task is not depend on the language.

Overview of Task 1extraction of complaint and diagnosis Task

(Shortly, NER task)

Two types of NER Task

• Given a raw text, find a disease name

• Given a raw text, find a disease name & its modality

腹痛は認めず腹痛は認められず

腹痛は認めず腹痛は認められず腹痛は認められず

Negative

Stomachache is not foundStomachache is not found

Stomachache is not foundStomachache is not found Stomachache is not found

(1) NER ONLY

(2) NER + MODALITY

MedNLP-1 << MedNLP-2

MedNLP-1 (2011) MedNLP-2 (2014)

15 groups over baseline


Seemingly MedNLP-2 much improved

MedNLP-1 << MedNLP-2MedNLP-1 (2011) MedNLP-2 (2014)



The accuracy of the best did not improve! Still 85% is the maxim→ we need a breakthrough

On the other hand, the average performance much improved.That shows participants have already learned the best way from the previous MedNLP-1, and used it→ We could successfully improve the level of NLP in this field

STILL, we can improvemodality detection

• In modality detection, we could see divergence in performance

• Several systems suffer from negation.

• Especially, detection of suspicion is difficult. and the half of systems (F-measure) is lower than 50%.

• The next challenge of this task is how to deal with such rare modalities

Overview of Task 2ICD coding task

(shortly coding task)

ICD-Coding Task2 ways to join

• Given a text with disease name, to give ICD-code to them

• Given a text without any information, to find a disease name, and give ICD-code to them

(1) TASK2ONLY

(2) TOTAL TASK

Divergence in performance

30%

70%

Difference is 40 %Rare case in recent shared task

Much Divergence in Task2ONLY

Difference is 50 %

30%

80%

Because

• Everything is unknown in new task–What kind of tool or method is good?• Supervised or un-supervised

–What kind of resource is good?• Extra corpus• Disease name Dictionary

–What is the “ICD-Coding” task all about?• Multi labeling• Document classification• Term similarity design

MethodsGroup

Method Tool Resource Approach

B RNN word2vec MEDIS Hyojun Byomei MasterICD-10 English dictionary

Supervised

C SVMBrown clustering

word2vec Wikipedia Supervised

D Distance in ICD tree hierarchy

MEDIS Hyojun Byomei Master -

E Full-test search LuceneGoogle translate

MEDIS Hyojun Byomei MasterICD-10 English dictionary

Unsupervised

F Pattern match MEDIS Hyojun Byomei Master Similarity Design

G Pattern matchBrown clustering

Unsupervised

H Logistic regression MEDIS Hyojun Byomei MasterLSD, T-Jisyo, MeDRA/J

-

J Rule MEDIS Hyojun Byomei Master Rule

K Full-text search, Exact match

Apache Solr

Unsupervised

MethodsGroup



Supervised







Unsupervised



Unsupervised


-



Apache Solr

Unsupervised

Much varieties in tool and methods, includingthe state-of-art tools, such as word2vec, RNN, are utilized

Interesting approach (using English resources using machine translation ) is utilized

The popular resource is “MEDIS Hyojun Byomei Master”BUT: half of groups do not use it

STILL: rule-based approach is employed

MethodsGroup



Supervised







Unsupervised



Unsupervised


-



Apache Solr

Unsupervised

We’d like to discuss the detail in the MedNLP Session (will be held day after tomorrow morning (9:20-)) Please join us

Overview of Task 3- Free Task --

What is Free Task?

• MedNLP has a unique task, FREE task, in which participants design their tasks freely (Any task is welcome!)

• We design this task because we are frequently asked “We’d like to join MedNLP. But, MedNLP task is NOT our target task” or “We could not have enough ability to develop the NLP systems”

• In order to save such groups, we proposed this task

• However, the "Free task" is much too open-ended– An NTCIR reviews said “I'm a little pessimistic about

whether anything concrete will come of this.”

• I am not so pessimistic, because 2 groups joined this task, presented interesting works.

F-group（Word Dictionary for Patients）

L-group（ Investigation Dictionary

Coverage）

ATOK covers the corpus?Several medical terms are too difficult,

and hard to understand for non-medical people, including patients and NLP researchers.To help the understanding of medical word, they build a word dictionary for non-medical people

Conclusion

SummaryMedNLP-1 MedNLP-2

Corpus Amount 50 documents 150 documentsMaterial Dummy Records Dummy Records

Medical Doctor Exam.Task De-identification

NERFree

NERICD-codingFree

# of systems 12 groupｓ(27 systems)

12(45 systems)

MedNLP-2 improvedProviding larger corpusDesigning more complex taskAlthough the number of groups is the same, but the number systems increased

AdviserMASUICHI Hiroshi, Ph.D.

AnnotatorSHIKATA ShukoKUBO KaySHIMAMOTO Yumiko

Acknowledgment

Eiji Aramaki (Kyoto university) Mizuki Morita (The university of Tokyo) Tomoko Ohkuma (Fuji Xerox)...

Documents

Transcript of Eiji Aramaki (Kyoto university) Mizuki Morita (The university of Tokyo) Tomoko Ohkuma (Fuji Xerox)...