1–11

CheXphoto: 10,000+ Photos and Transformations of ChestX-rays for Benchmarking Deep Learning Robustness

Nick A. Phillips∗ nphill22@stanford.edu

Pranav Rajpurkar∗ pranavsr@stanford.edu

Mark Sabini∗ msabini@alumni.stanford.edu

Rayan Krishnan rayank@stanford.edu

Sharon Zhou sharonz@cs.stanford.edu

Anuj Pareek anujpare@stanford.edu

Nguyet Minh Phu minhphu@stanford.edu

Chris Wang chrwang@stanford.edu

Mudit Jain muditjai@gmail.com

Nguyen Duong Du v.dunguyen@vinbrain.net

Steven QH Truong Brain01@vinbrain.net

Andrew Y. Ng ang@cs.stanford.edu

Matthew P. Lungren mlungren@stanford.edu

Abstract

Clinical deployment of deep learningalgorithms for chest x-ray interpreta-tion requires a solution that can inte-grate into the vast spectrum of clini-cal workflows across the world. An ap-pealing approach to scaled deploymentis to leverage the ubiquity of smart-phones by capturing photos of x-raysto share with clinicians using messagingservices like WhatsApp. However, theapplication of chest x-ray algorithms tophotos of chest x-rays requires reliableclassification in the presence of arti-facts not typically encountered in dig-ital x-rays used to train machine learn-ing models. We introduce CheXphoto,a dataset of smartphone photos andsynthetic photographic transformationsof chest x-rays sampled from the CheX-pert dataset. To generate CheXphotowe (1) automatically and manually cap-tured photos of digital x-rays under dif-ferent settings, and (2) generated syn-thetic transformations of digital x-rays

∗ These authors contributed equally to this work

targeted to make them look like pho-tos of digital x-rays and x-ray films.We release this dataset as a resourcefor testing and improving the robust-ness of deep learning algorithms for au-tomated chest x-ray interpretation onsmartphone photos of chest x-rays.

1. Background & Summary

Chest x-rays are the most common imag-ing exams, critical for diagnosis and man-agement of many diseases and medical pro-cedures. With over 2 billion chest x-raysperformed globally each year, many clinicsin both developing and developed countrieshave an insufficient number of radiologists toperform timely x-ray interpretation [Witheyet al., 2019, Andronikou et al., 2011]. Com-puter algorithms could help reduce the short-age for x-ray interpretation worldwide.

Several recent advances in training deeplearning algorithms for automated chest x-ray interpretation have been made possible

© N. A. Phillips et al.

arX

iv:2

007.

0619

9v2

[ee

ss.I

V]

11

Dec

202

0

CheXphoto

Figure 1: Overview of the CheXphoto data generation process.

by large datasets [Hwang et al., 2019, Singhet al., 2018]. In controlled settings, thesedeep learning algorithms can learn from la-beled data to automatically detect patholo-gies at an accuracy comparable to thatof practicing radiologists [Kallianos et al.,2019]. These developments have been fu-eled by both improvements in deep learn-ing algorithms for image classification tasks[Huang et al., 2016, Rajpurkar et al., 2018],and by the release of large public datasets[Irvin et al., 2019, Wang et al., 2017, Bus-tos et al., 2019, Johnson et al., 2019]. Al-though these algorithms have demonstratedthe potential to provide accurate chest x-rayinterpretation and increase access to radi-ology expertise, major obstacles remain intheir translation to the clinical setting [Namet al., 2019, Singh et al., 2018].

One significant obstacle to the adoption ofchest x-ray algorithms is that deployment re-quires a solution that can integrate into thevast spectrum of clinical workflows aroundthe world. Most chest x-ray algorithmsare developed and validated on digital x-rays, while the majority of developing regionsuse films [Schwartz et al., 2014, Andronikou

et al., 2011]. An appealing approach toscaled deployment is to leverage the ubiq-uity of existing smartphones: automatedinterpretation of x-ray film through cellphone photography has emerged through a“store-and-forward telemedicine” approach,in which one or more digital photos of chestfilms are sent as email attachments or instantmessages by practitioners to obtain secondopinions from specialists as part of clinicalcare [Goost et al., 2012, Vassallo et al., 1998].Furthermore, studies have shown that pho-tographs of films using modern phone cam-eras are of equivalent diagnostic quality tothe films themselves [Schwartz et al., 2014],indicating the feasibility of high-quality au-tomated algorithmic interpretation of photosof x-ray films.

Automated interpretation of chest x-rayphotos at the same level of performance asdigital chest x-rays is challenging becausephotography introduces visual artifacts notcommonly found in digital x-rays, such asaltered viewing angles, variable ambient andbackground lighting conditions, glare, moire,rotations, translations, and blur [Kurakinet al., 2017]. Image classification algorithms

2

CheXphoto

have been shown to experience a significantdrop in performance when input images areperceived through a camera [Kurakin et al.,2017]. Although recent work has demon-strated good generalizability of deep learningalgorithms trained on digital x-rays to pho-tographs [Rajpurkar et al., 2020], interpreta-tion performance could be improved throughinclusion of x-ray photography in the train-ing process [Hendrycks and Dietterich, 2019,Goodfellow et al., 2014]. However, there arecurrently no large-scale public datasets ofphotos of chest x-rays.

To meet this need, we developed CheX-photo, a dataset of photos of chest x-raysand synthetic transformations designed tomimic the effects of photography. We be-lieve that CheXphoto will enable researchersto improve and evaluate model performanceon photos of x-rays, reducing the barrier toclinical deployment.

2. Methods

We introduce CheXphoto, a dataset of pho-tos of chest x-rays and synthetic transforma-tions designed to mimic the effects of pho-tography. Specifically, CheXphoto includes aset of (1) Natural Photos: automatically andmanually captured photos of x-rays underdifferent settings, including various lightingconditions and locations, and (2) SyntheticTransformations: targeted transformationsof digital x-rays to simulate the appearanceof photos of digital x-rays and x-ray films.The x-rays used in CheXphoto are primarilysampled from CheXpert, a large dataset of224,316 chest x-rays of 65,240 patients, withassociated labels for 14 observations from ra-diology reports [Irvin et al., 2019].

CheXphoto comprises a training set of nat-ural photos and synthetic transformations of10,507 x-rays from 3,000 unique patients thatwere sampled at random from the CheXperttraining set, and validation and test sets of

natural and synthetic transformations of all234 x-rays from 200 patients and 668 x-raysfrom 500 patients in the CheXpert valida-tion and test sets, respectively. In addition,the CheXphoto validation set includes 200natural photos of physical x-ray films sam-pled from external data sources, intended tomore closely simulate pictures taken by radi-ologists in developing world clinical settings.As much of the developing world performs x-ray interpretation on film, this distinct set ofimages enables users to perform additionalvalidation on a novel task that may be en-countered in clinical deployment.

2.1. Acquiring Natural Photos ofChest X-Rays

Natural photos consist of x-ray photographyusing cell phone cameras in various lightingconditions and environments. We developedtwo sets of natural photos: images capturedthrough an automated process using a Nokia6.1 cell phone, and images captured manu-ally with an iPhone 8.

2.1.1. Automated Capture ofNokia10k dataset

We developed the ‘Nokia10k’ dataset by cap-turing 10,507 images of digital chest x-raysusing a tripod-mounted Nokia 6.1 cell phone(16 megapixel camera with a Zeiss sensor)and a custom Android application termedCheXpeditor to fully automate the processesof photography and metadata management.The primary challenge in automation wassynchronizing picture-taking on the phonewith displaying the chest x-ray on the moni-tor, to maintain a 1-to-1 correspondence be-tween each chest x-ray and its photographedimage. Without a robust synchronizationmethod, photos of chest x-rays might beskipped or duplicated, jeopardizing the datacollection process. Thus, bidirectional com-munication over UDP was established be-

3

CheXphoto

Phone

UDP Server App UI

Storage

Computer

UDP Client

Storage

Monitor Camera

1

2

3

4

6

7

5

a.

b. c.

Figure 2: Acquiring Natural Photos of Chest X-Rays Using Automated Capture a. Visualrepresentation of the automated picture-taking process used for Nokia10k. The steps aredescribed: 1. X-ray retrieved from computer storage, 2. X-ray displayed on monitor,3. X-ray index and metadata sent to phone over UDP, 4. Index verified by phone, andcamera triggered, 5. Application UI updated with new picture and filename, 6. Picturesaved to phone storage with metadata in filename, 7. Computer notified that imaging wassuccessful. b. The physical setup used for Nokia10k, set in an example environment. c.Phone application UI, displaying most recent picture and saved filename.

tween the phone and the computer drivingthe monitor to exchange image metadata,take photos, and advance the chest x-ray onthe monitor.

The 10,507 x-rays in Nokia10k were in-dexed deterministically from 1 to N . Weselected disjoint subsets of 250 to 500 consec-utive indices to be photographed in constantenvironmental conditions. For each subset of

indices, photography was conducted as fol-lows:

1) The ith chest x-ray was retrieved fromcomputer storage. 2) The ith chest x-raywas displayed on the monitor. 3) The imagemetadata m was assembled by the computer,and (i,m) were sent to the phone via UDP.4) The phone verified that i was one greaterthan the previous index. If so, its camerawas triggered. Else, the computer was noti-fied of an error, and the entire picture-taking

4

CheXphoto

Table 1: The distribution of labeled observations for the Nokia10k training dataset.

Pathology Positive (%) Uncertain (%) Negative (%)

No Finding 972 (9.25) 0 (0.00) 9535 (90.75)Enlarged Cardiomediastinum 518 (4.93) 600 (5.71) 9389 (89.36)Cardiomegaly 1313 (12.50) 370 (3.52) 8824 (83.98)Lung Opacity 5184 (49.34) 213 (2.03) 5110 (48.63)Lung Lesion 415 (3.95) 78 (0.74) 10014 (95.31)Edema 2553 (24.30) 634 (6.03) 7320 (69.67)Consolidation 671 (6.39) 1315 (12.52) 8521 (81.10)Pneumonia 263 (2.50) 885 (8.42) 9359 (89.07)Atelectasis 1577 (15.01) 1595 (15.18) 7335 (69.81)Pneumothorax 957 (9.11) 166 (1.58) 9384 (89.31)Pleural Effusion 4115 (39.16) 607 (5.78) 5785 (55.06)Pleural Other 170 (1.62) 127 (1.21) 10210 (97.17)Fracture 391 (3.72) 31 (0.30) 10085 (95.98)Support Devices 5591 (53.21) 48 (0.46) 4868 (46.33)

process was aborted. 5) The phone appli-cation UI, responsible for displaying statusand current image, was updated to show thenew picture and filename. 6) The picturewas saved to phone storage with the meta-data m embedded in the filename. 7) Thephone notified the computer that the imag-ing was successful, and the entire process wasrepeated for the i + 1st chest x-ray.

After all images for a Nokia10k subset weretaken, they were exported in one batch fromthe phone to storage. The metadata wasparsed from the image filenames and usedto automatically assign the correct CheX-pert label. Alterations made to the imagingconditions after every subset included mov-ing to a different room, switching the roomlight on/off, opening/closing the window-blinds, rotating the phone orientation be-tween portrait/landscape, adjusting the po-sition of the tripod, moving the mouse cur-sor, varying the monitor’s color temperature,and switching the monitor’s screen finish be-tween matte/glossy. In all conditions, thechest x-ray was centered in the camera view-

finder and lung fields were contained withinthe field of view.

2.1.2. Manual Capture of iPhone1kdataset

We developed the ‘iPhone1k dataset’ bymanually capturing 1,000 images of digitalchest x-rays using an iPhone 8 (12 megapixelcamera with a Sony Exmor RS sensor).The digital x-rays selected for the iPhone1kdataset are a randomly sampled subset of thex-rays used in the Nokia10k dataset. To pro-duce the iPhone1k dataset, chest x-rays weredisplayed in full-screen on a computer moni-tor with 1920 x 1080 screen resolution and ablack background. A physician took photosof the chest x-rays with a handheld iPhone 8using the standard camera app. The physi-cian was advised to change angle and dis-tance from the computer monitor in-betweeneach picture within constraints.

For all images, the chest x-ray was cen-tered in the viewfinder of the camera, andthe thoracic and lung fields were containedwithin the field of view. Conformant to ra-

5

CheXphoto

diological chest x-ray standards, both lung-apices and costodiaphragmatic recesses wereincluded craniocaudally, and the edges of theribcage were included laterally. Photos werecaptured in sets of 100 to 200 images at atime; between sets, ambient alterations weremade, such as switching the room-lightingon/off, opening or closing of the window-blinds, and physically moving the computermonitor to a different location in the room.

2.2. Generating SyntheticPhotographic Transformations ofChest X-Rays

Synthetic transformations consist of auto-mated changes to digital x-rays designed tosimulate the appearance of photos of digi-tal x-rays and x-ray films. We developedtwo sets of complementary synthetic trans-formations: digital transformations to altercontrast and brightness, and spatial trans-formations to add glare, moire effects andperspective changes. To ensure that the levelof these transformations did not impact thequality of the image for physician diagnosis,the images were verified by a physician. Insome cases, the effects may be visually im-perceptible, but may still be adversarial forclassification models. For both sets, we applythe transformations to the same 10,507 digi-tal x-rays selected for the Nokia10k dataset.

Digital transformations were produced bysuccessive random alterations of contrast andbrightness. First, the image was either en-hanced for greater or lesser contrast. Set-ting a contrast factor of 1 for the originalimage, the contrast up transformation in-creased contrast by a factor of 1.1 and thecontrast down transformation decreased con-trast by a factor of 0.83. For both these fac-tors, random noise between -0.01 and 0.01was applied. After the contrast modification,the brightness of the image was then trans-formed randomly up or down using the same

numeric factors. Both the brightness andcontrast transformations were applied usingthe Python PIL ImageEnhance class.

Spatial transformations consisted of alter-ations to add glare, moire effects and per-spective changes. First, we applied a glarematte transformation to simulate the effectof photographing a glossy film which reflectsambient light. This was produced by ran-domly generating a two-dimensional multi-variate normal distribution which describesthe location of a circular, white mask. Sec-ond, a moire effect was added to simulatethe pattern of parallel lines seen by digi-tal cameras when taking pictures of com-puter screens. The effect is produced as aresult of the difference in rates of shutterspeed and LCD screen sweeping refresh rate.The moire effect was simulated by generat-ing semi-transparent parallel lines, warpingthem and overlaying them onto each image.Finally, a tilt effect was added to simulaterandom distortions of perspective that mayarise in taking a photo at angle to the screen.The tilt effect was produced by randomlyscaling the x and y values of each of the cor-ners by a factor between 0 and 0.05 towardsthe center. This random movement of cor-ners is used to skew the entire photo.

Both the digital and spatial transforma-tions are provided in CheXphoto. Eachtransformation may be reproduced individ-ually using the code provided. Additionaltransformations - glare glossy, blur, motion,rotation, translation - are also included.

2.3. Validation and Test

We developed a CheXphoto validation andtest set to be used for model validation andevaluation. The validation set comprises nat-ural photos and synthetic transformations ofall 234 x-rays in the CheXpert validation set,and is included in the public release, whilethe test set comprises natural photos of all

6

CheXphoto

Table 2: Natural Photos (a-b) and Synthetic Transformations (Digital (c-f) and Spatial(g-i)) included in CheXphoto.

(a) iPhone (b) Nokia

(c) Brightness Up (d) Brightness Down (e) Contrast Up (f) Contrast Down

(g) Glare Matte (h) Moire (i) Tilt

668 x-rays in the CheXpert test set, and iswithheld for evaluation purposes.

We generated the natural photos of thevalidation set by manually capturing imagesof x-rays displayed on a 2560 × 1080 monitorusing a OnePlus 6 cell phone (16 megapixelcamera with a Sony IMX 519 sensor), follow-ing a protocol that mirrored the iPhone1kdataset. Synthetic transformations of thevalidation images were produced using thesame protocol as the synthetic training set.The test set was captured using an iPhone 8,following the same protocol as the iPhone1kdataset.

The validation set contains an additional200 cell phone photos of x-ray films for 200unique patients. As photos of physical x-rayfilms, this component of the validation set isdistinct from the previously described nat-

ural and synthetic transformations of digi-tal x-rays. Films for 119 patients were sam-pled from the MIMIC-CXR dataset [Johnsonet al., 2019], and films for 81 patients wereprovided by VinBrain, a subsidiary of Vin-group in Vietnam, and originally collectedthrough joint research projects with leadinglung hospitals in Vietnam. The film datasetspans 5 observation labels (atelectasis, car-diomegaly, consolidation, edema, pleural ef-fusion), with 40 images supporting each ob-servation. Observation labels for each imagewere manually verified by a physician. Im-ages were captured using a VinSmart phonewith a 12MP camera by positioning the phys-ical x-ray film vertically on a light box in typ-ical clinical lighting conditions, and imageswere automatically cropped and oriented.

7

CheXphoto

Table 3: The number of patients, studies,and images in CheXphoto.

Dataset Patients Studies Images

TrainingiPhone 295 829 1,000Nokia 3,000 8,931 10,507Synthetic 3,000 8,931 10,507

ValidationNatural 200 200 234Synthetic 200 200 234Film 200 200 200

Test 500 500 668

Figure 3: CheXphoto directory structure

2.4. Technical Validation

CheXphoto was developed using images andlabels from the CheXpert dataset [Irvinet al., 2019]. Photography of x-rays wasconducted in a controlled setting in accor-dance with the protocols documented in theMethods section, which were developed withphysician consultation. Although CheX-photo contains multiple images for some pa-tients, either from the same or different stud-ies, there is no patient overlap between thetraining, validation, and test sets. Code de-veloped for synthetic transformations is ver-sion controlled and made available as anopen source resource for review and modi-fication. All images are uniquely identifiableby patient ID, study ID, and view, and theage and sex of each patient is provided inthe data description CSV file. The original,unaltered images can be obtained from theCheXpert dataset by the unique identifiers.

The CheXphoto dataset is organized byby transformation; the training and valida-tion sets contain directories corresponding tothe method of data generation. Within eachdirectory, the x-ray images are organized insubdirectories by a patient identifier, studyID, and one or more individual views. Im-ages are stored as JPEG files, and image di-

mensions vary according to the method ofgeneration. Each transformation set has anassociated CSV file, which provides observa-tion labels from the CheXpert dataset andrelative paths to the corresponding images.

2.5. Data Access

The CheXphoto training and validation setsare available for download1. The CheXphototest set is withheld for official evaluation ofmodels. CheXphoto users may submit theirexecutable code, which is then run on theprivate test set, preserving the integrity ofthe test results. The testing process is en-abled by CodaLab [Liang et al., 2020], anonline platform for collaborative and repro-ducible computational research. CodaLabWorksheets exposes a simple command-lineinterface, which enables users to submit aDocker image, dependencies, and the neces-sary commands to run their models. Thesefeatures allow us to run arbitrary code sub-missions on the withheld test set. Once auser has successfully uploaded their code toCodaLab, we will evaluate their performanceon the withheld test set and share results ona live leaderboard on the web.

1. https://stanfordmlgroup.github.io/competitions/chexphoto

8

CheXphoto

In addition, the code used to preparethe synthetically generated dataset is pub-licly available2. The synthetic transforma-tions can be reproduced by running thesynthesize.py script with the appropriateCSV file containing the paths to the imagesfor which the perturbation is to be applied.Detailed instructions on flags and usage areincluded in the repository README.

3. Conclusion

We believe that CheXphoto will enablegreater access to automated chest x-rayinterpretation algorithms worldwide, prin-cipally in healthcare systems that arepresently excluded from the benefits of dig-ital medicine. By facilitating the develop-ment, validation, and testing of automatedchest x-ray interpretation algorithms with aubiquitous technology such as smartphonephotography, CheXphoto broadens access tointerpretation algorithms in underdevelopedregions, where this technology is poised tohave the greatest impact on the availabilityand quality of healthcare.

References

Savvas Andronikou, Kieran McHugh, Nu-raan Abdurahman, Bryan Khoury, Vic-tor Mngomezulu, William E. Brant,Ian Cowan, Mignon McCulloch, andNathan Ford. Paediatric radiology seenfrom africa. part i: providing diag-nostic imaging to a young population.Pediatric Radiology, 41(7):811–825, Jul2011. ISSN 1432-1998. doi: 10.1007/s00247-011-2081-8. URL https://doi.

org/10.1007/s00247-011-2081-8.

Aurelia Bustos, Antonio Pertusa, Jose-MariaSalinas, and Maria de la Iglesia-Vaya. Pad-

2. https://github.com/stanfordmlgroup/cheXphoto

chest: A large chest x-ray image datasetwith multi-label annotated reports, 2019.

Ian J. Goodfellow, Jonathon Shlens, andChristian Szegedy. Explaining and har-nessing adversarial examples, 2014.

Hans Goost, Johannes Witten, AndreasHeck, Dariusch Hadizadeh, Oliver Weber,Ingo Graff, Christof Burger, Mareen Mon-tag, Felix Koerfer, and Koroush Kabir.Image and diagnosis quality of x-ray im-age transmission via cell phone camera: Aproject study evaluating quality and reli-ability. PloS one, 7:e43402, 10 2012. doi:10.1371/journal.pone.0043402.

Dan Hendrycks and Thomas G. Dietterich.Benchmarking neural network robustnessto common corruptions and perturbations.CoRR, abs/1903.12261, 2019. URL http:

//arxiv.org/abs/1903.12261.

Gao Huang, Zhuang Liu, and Kilian Q.Weinberger. Densely connected convolu-tional networks. CoRR, abs/1608.06993,2016. URL http://arxiv.org/abs/

1608.06993.

Eui Jin Hwang, Sunggyun Park, Kwang-Nam Jin, Jung Im Kim, So Young Choi,Jong Hyuk Lee, Jin Mo Goo, JaehongAum, Jae-Joon Yim, Julien G. Cohen,Gilbert R. Ferretti, Chang Min Park,for the DLAD Development, and Eval-uation Group. Development and Val-idation of a Deep Learning–Based Au-tomated Detection Algorithm for Ma-jor Thoracic Diseases on Chest Radio-graphs. JAMA Network Open, 2(3):e191095–e191095, 03 2019. ISSN 2574-3805. doi: 10.1001/jamanetworkopen.2019.1095. URL https://doi.org/10.

1001/jamanetworkopen.2019.1095.

Jeremy Irvin, Pranav Rajpurkar, MichaelKo, Yifan Yu, Silviana Ciurea-Ilcus, Chris

9

CheXphoto

Chute, Henrik Marklund, Behzad Hagh-goo, Robyn L. Ball, Katie S. Shpan-skaya, Jayne Seekins, David A. Mong,Safwan S. Halabi, Jesse K. Sandberg,Ricky Jones, David B. Larson, Curtis P.Langlotz, Bhavik N. Patel, Matthew P.Lungren, and Andrew Y. Ng. Chexpert:A large chest radiograph dataset with un-certainty labels and expert comparison.CoRR, abs/1901.07031, 2019. URL http:

//arxiv.org/abs/1901.07031.

Alistair E. W. Johnson, Tom J. Pollard,Seth J. Berkowitz, Nathaniel R. Green-baum, Matthew P. Lungren, Chih-yingDeng, Roger G. Mark, and Steven Horng.Mimic-cxr, a de-identified publicly avail-able database of chest radiographs withfree-text reports. Scientific Data, 6(1):317, Dec 2019. ISSN 2052-4463. doi: 10.1038/s41597-019-0322-0. URL https://

doi.org/10.1038/s41597-019-0322-0.

K. Kallianos, J. Mongan, Sameer Antani,T. Henry, A. Taylor, Joseph Abuya, andM. Kohli. How far have we come? artifi-cial intelligence for chest radiograph inter-pretation. Clinical Radiology, 74, 05 2019.doi: 10.1016/j.crad.2018.12.015.

Alexey Kurakin, Ian Goodfellow, and SamyBengio. Adversarial examples in the phys-ical world. ICLR Workshop, 2017. URLhttps://arxiv.org/abs/1607.02533.

Percy Liang, Isabelle Guyon, Evelyne Vie-gas, Sergio Escalera, Xavier Baro Sole, andEric Carmichael. CodaLab, 2020. https:

//codalab.org.

Ju Gang Nam, Sunggyun Park, Eui JinHwang, Jong Hyuk Lee, Kwang-NamJin, Kun Young Lim, Thienkai Huy Vu,Jae Ho Sohn, Sangheum Hwang, Jin MoGoo, and Chang Min Park. Devel-opment and validation of deep learn-ing–based automatic detection algorithm

for malignant pulmonary nodules onchest radiographs. Radiology, 290(1):218–228, 2019. doi: 10.1148/radiol.2018180237. URL https://doi.org/

10.1148/radiol.2018180237. PMID:30251934.

Pranav Rajpurkar, Jeremy Irvin, Robyn L.Ball, Kaylie Zhu, Brandon Yang, HershelMehta, Tony Duan, Daisy Ding, AartiBagul, Curtis P. Langlotz, Bhavik N. Pa-tel, Kristen W. Yeom, Katie Shpanskaya,Francis G. Blankenberg, Jayne Seekins,Timothy J. Amrhein, David A. Mong,Safwan S. Halabi, Evan J. Zucker, An-drew Y. Ng, and Matthew P. Lungren.Deep learning for chest radiograph di-agnosis: A retrospective comparison ofthe chexnext algorithm to practicing ra-diologists. PLOS Medicine, 15(11):1–17, 11 2018. doi: 10.1371/journal.pmed.1002686. URL https://doi.org/10.

1371/journal.pmed.1002686.

Pranav Rajpurkar, Anirudh Joshi, AnujPareek, Phil Chen, Amirhossein Kiani,Jeremy Irvin, Andrew Y. Ng, andMatthew P. Lungren. CheXpedition: In-vestigating Generalization Challenges forTranslation of Chest X-Ray Algorithms tothe Clinical Setting. arXiv e-prints, art.arXiv:2002.11379, February 2020.

Adam B Schwartz, Gina Siddiqui, John SBarbieri, Amana L Akhtar, Woojin Kim,Ryan Littman-Quinn, Emily F Conant,Narainder K Gupta, Bryan A Puke-nas, Parvati Ramchandani, Anna S Lev-Toaff, Jennifer D Tobey, Drew A Tori-gian, Amy H Praestgaard, and Carrie LKovarik. The accuracy of mobile tel-eradiology in the evaluation of chest x-rays. Journal of Telemedicine and Tele-care, 20(8):460–463, 2014. doi: 10.1177/1357633X14555639. URL https://doi.

10

CheXphoto

org/10.1177/1357633X14555639. PMID:25322696.

Ramandeep Singh, Mannudeep K. Kalra,Chayanin Nitiwarangkul, John A. Patti,Fatemeh Homayounieh, Atul Padole,Pooja Rao, Preetham Putha, Victorine V.Muse, Amita Sharma, and Subba R. Digu-marthy. Deep learning in chest radio-graphy: Detection of findings and pres-ence of change. PLOS ONE, 13(10):1–12, 10 2018. doi: 10.1371/journal.pone.0204155. URL https://doi.org/

10.1371/journal.pone.0204155.

DJ Vassallo, PJ Buxton, JH Kilbey, andM Trasler. The first telemedicine link forthe british forces. BMJ Military Health,144(3):125–130, 1998. ISSN 0035-8665.doi: 10.1136/jramc-144-03-02. URLhttps://militaryhealth.bmj.com/

content/144/3/125.

X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri,and R. M. Summers. Chestx-ray8:Hospital-scale chest x-ray database andbenchmarks on weakly-supervised classi-fication and localization of common tho-rax diseases. In 2017 IEEE Conference onComputer Vision and Pattern Recognition(CVPR), pages 3462–3471, 2017.

Samuel Withey, Vicky Goh, GiovanniMontana, Emanuele Pesce, and RobertBakewell. Automated triaging of adultchest radiographs with deep artificial neu-ral networks. Radiology, 2019:1–7, 01 2019.doi: 10.1148/radiol.2018180921.

11