george@gs.ai

Machine Learning Annotations

September 10, 2018

George Shih, MDCo-chair SIIM ML Committee

Associate Professor and Vice-Chair for Informatics Weill Cornell Radiology

george@gs.ai

Disclosures

george@gs.ai

https://hackernoon.com/%EF%B8%8F-big-challenge-in-deep-learning-training-data-31a88b97b282

george@gs.ai

george@gs.ai

15,000 exams from CXR14 entire dataset.

- all 5,300 pneumothorax

- random 5,000 normal

- random 5,000 not pneumothorax, not normal

crowdsourcing annotations

george@gs.ai

george@gs.ai

Crowdsourcing Pneumothorax Annotations using Machine Learning Annotations (MLA) on the NIH Chest X-ray DatasetSrinidhi G. Myadaboyina, MS, New York University, MD.ai

Yashovardhan Chaturvedi; Anouk Stein, MD; Leon Chen, MD; Stephen Borstelmann, MD; Ross Filice, MD; Paras C. Lakhani, MD; Maansi Parekh, MD; Prasanth Prasanna, MD; Alexandre Cadrin-Chênevert, MD; George Shih, MD, MS

george@gs.ai

Sources of Annotations

Human (radiologist) annotations

ML algorithms for generating annotations

Any ML algorithm output = annotations?

george@gs.ai

Standardizing AI Annotations - The DICOM Way

David A. Clunie, MBBS, FSIIM, PixelMed Publishing

Enabling Automated Search for Relevant Machine Learning Datasets

Steve G. Langer, PhD, CIIP, Mayo Clinic Rochester

Image Labeling for Deep Learning: Machine vs. Human

Curtis P. Langlotz, MD, PhD, FSIIM, Stanford University

Scaling up Image Annotation for Deep Learning: Standards, Labels from

Text, and Leveraging Multi-institutional Data

Daniel L. Rubin, MD, MS, FSIIM, Stanford University

Panel Discussion

Luminary Panel