An overview of deep learning in medical imagingfocusing on MRI

Alexander Selvikvag Lundervolda,b,∗, Arvid Lundervolda,c,d

aMohn Medical Imaging and Visualization Centre (MMIV), Haukeland University Hospital, NorwaybDepartment of Computing, Mathematics and Physics, Western Norway University of Applied Sciences, NorwaycNeuroinformatics and Image Analysis Laboratory, Department of Biomedicine, University of Bergen, Norway

dDepartment of Health and Functioning, Western Norway University of Applied Sciences, Norway

Abstract

What has happened in machine learning lately, and what does it mean for the future of medicalimage analysis? Machine learning has witnessed a tremendous amount of attention over the lastfew years. The current boom started around 2009 when so-called deep artificial neural networksbegan outperforming other established models on a number of important benchmarks. Deep neuralnetworks are now the state-of-the-art machine learning models across a variety of areas, from imageanalysis to natural language processing, and widely deployed in academia and industry. Thesedevelopments have a huge potential for medical imaging technology, medical data analysis, medicaldiagnostics and healthcare in general, slowly being realized. We provide a short overview of recentadvances and some associated challenges in machine learning applied to medical image processingand image analysis. As this has become a very broad and fast expanding field we will not surveythe entire landscape of applications, but put particular focus on deep learning in MRI.

Our aim is threefold: (i) give a brief introduction to deep learning with pointers to core refer-ences; (ii) indicate how deep learning has been applied to the entire MRI processing chain, fromacquisition to image retrieval, from segmentation to disease prediction; (iii) provide a starting pointfor people interested in experimenting and perhaps contributing to the field of machine learning formedical imaging by pointing out good educational resources, state-of-the-art open-source code, andinteresting sources of data and problems related medical imaging.

Keywords: Machine learning, Deep learning, Medical imaging, MRI

1. Introduction

Machine learning has seen some dramatic developments recently, leading to a lot of interest fromindustry, academia and popular culture. These are driven by breakthroughs in artificial neuralnetworks, often termed deep learning, a set of techniques and algorithms that enable computersto discover complicated patterns in large data sets. Feeding the breakthroughs is the increasedaccess to data (“big data”), user-friendly software frameworks, and an explosion of the availablecompute power, enabling the use of neural networks that are deeper than ever before. Thesemodels nowadays form the state-of-the-art approach to a wide variety of problems in computervision, language modeling and robotics.

Deep learning rose to its prominent position in computer vision when neural networks startedoutperforming other methods on several high-profile image analysis benchmarks. Most famously

∗Corresponding authorEmail addresses: allu@hvl.no (Alexander Selvikvag Lundervold), mfyal@uib.no (Arvid Lundervold)

Preprint submitted to Zeitschrift fur Medizinische Physik December 18, 2018

arX

iv:1

811.

1005

2v2

[cs

.CV

] 1

6 D

ec 2

018

on the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC)1 in 2012 [1] when a deeplearning model (a convolutional neural network) halved the second best error rate on the imageclassification task. Enabling computers to recognize objects in natural images was until recentlythought to be a very difficult task, but by now convolutional neural networks have surpassed evenhuman performance on the ILSVRC, and reached a level where the ILSVRC classification task isessentially solved (i.e. with error rate close to the Bayes rate). The last ILSVRC competitionwas held in 2017, and computer vision research has moved on to other more difficult benchmarkchallenges. For example the Common Objects in Context Challenge (COCO) [2].

Deep learning techniques have become the de facto standard for a wide variety of computer visionproblems. They are, however, not limited to image processing and analysis but are outperformingother approaches in areas like natural language processing [3, 4, 5], speech recognition and synthesis[6, 7]2, and in the analysis of unstructured, tabular-type data using entity embeddings [8, 9].3

The sudden progress and wide scope of deep learning, and the resulting surge of attention andmulti-billion dollar investment, has led to a virtuous cycle of improvements and investments inthe entire field of machine learning. It is now one of the hottest areas of study world-wide [15],and people with competence in machine learning are highly sought-after by both industry andacademia4.

Healthcare providers generate and capture enormous amounts of data containing extremelyvaluable signals and information, at a pace far surpassing what “traditional” methods of analysiscan process. Machine learning therefore quickly enters the picture, as it is one of the best waysto integrate, analyze and make predictions based on large, heterogeneous data sets (cf. healthinformatics [16]). Healthcare applications of deep learning range from one-dimensional biosignalanalysis [17] and the prediction of medical events, e.g. seizures [18] and cardiac arrests [19], tocomputer-aided detection [20] and diagnosis [21] supporting clinical decision making and survivalanalysis [22], to drug discovery [23] and as an aid in therapy selection and pharmacogenomics [24],to increased operational efficiency [25], stratified care delivery [26], and analysis of electronic healthrecords [27, 28].

The use of machine learning in general and deep learning in particular within healthcare is still inits infancy, but there are several strong initiatives across academia, and multiple large companies arepursuing healthcare projects based on machine learning. Not only medical technology companies,but also for example Google Brain [29, 30, 31]5, DeepMind [32]6, Microsoft [33, 34]7 and IBM [35]8.There is also a plethora of small and medium-sized businesses in the field9.

1Colloquially known as the ImageNet challenge2Try it out here: https://deepmind.com/blog/wavenet-generative-model-raw-audio3As a perhaps unsurprising side-note, these modern deep learning methods have also entered the field of physics.

Among other things, they are tasked with learning physics from raw data when no good mathematical models areavailable. For example in the analysis of gravitational waves where deep learning has been used for classification[10], anomaly detection [11] and denoising [12], using methods that are highly transferable across domains (thinkEEG and fMRI). They are also part of mathematical model and machine learning hybrids [13, 14], formed to reducecomputational costs by having the mathematical model train a machine learning model to perform its job, or toimprove the fit with observations in settings where the mathematical model can’t incorporate all details (think noise).

4See e.g. https://economicgraph.linkedin.com/research/LinkedIns-2017-US-Emerging-Jobs-Report for astudy focused on the US job market

5https://ai.google/research/teams/brain/healthcare-biosciences6https://deepmind.com/applied/deepmind-health/7https://www.microsoft.com/en-us/research/research-area/medical-health-genomics8https://www.research.ibm.com/healthcare-and-life-sciences9Aidoc, Arterys, Ayasdi, Babylon Healthcare Services, BenevolentAI, Enlitic, EnvoiAI, H2O, IDx, MaxQ AI,

Mirada Medical, Viz.ai, Zebra Medical Vision, and many more.

2

2. Machine learning, artificial neural networks, deep learning

In machine learning one develops and studies methods that give computers the ability to solveproblems by learning from experiences. The goal is to create mathematical models that can betrained to produce useful outputs when fed input data. Machine learning models are provided ex-periences in the form of training data, and are tuned to produce accurate predictions for the trainingdata by an optimization algorithm. The main goal of the models are to be able to generalize theirlearned expertise, and deliver correct predictions for new, unseen data. A model’s generalizationability is typically estimated during training using a separate data set, the validation set, and usedas feedback for further tuning of the model. After several iterations of training and tuning, the finalmodel is evaluated on a test set, used to simulate how the model will perform when faced with new,unseen data.

There are several kinds of machine learning, loosely categorized according to how the modelsutilize its input data during training. In reinforcement learning one constructs agents that learnfrom their environments through trial and error while optimizing some objective function. A famousrecent application of reinforcement learning is AlphaGo and AlphaZero [36], the Go-playing machinelearning systems developed by DeepMind. In unsupervised learning the computer is tasked withuncovering patterns in the data without our guidance. Clustering is a prime example. Most oftoday’s machine learning systems belong to the class of supervised learning. Here, the computeris given a set of already labeled or annotated data, and asked to produce correct labels on new,previously unseen data sets based on the rules discovered in the labeled data set. From a set ofinput-output examples, the whole model is trained to perform specific data-processing tasks. Imageannotation using human-labeled data, e.g. classifying skin lesions according to malignancy [37] ordiscovering cardiovascular risk factors from retinal fundus photographs [38], are two examples ofthe multitude of medical imaging related problems attacked using supervised learning.

Machine learning has a long history and is split into many sub-fields, of which deep learning isthe one currently receiving the bulk of attention.

There are many excellent, openly available overviews and surveys of deep learning. For shortgeneral introductions to deep learning, see [39, 40]. For an in-depth coverage, consult the freelyavailable book [41]10. For a broad overview of deep learning applied to medical imaging, see [42].We will only mention some bare essentials of the field, hoping that these will serve as useful pointersto the areas that are currently the most influential in medical imaging.

2.1. Artificial neural networks

Artificial neural networks (ANNs) is one of the most famous machine learning models, introducedalready in the 1950s, and actively studied since [41, Chapter 1.2].11

Roughly, a neural network consists of a number of connected computational units, called neurons,arranged in layers. There’s an input layer where data enters the network, followed by one or morehidden layers transforming the data as it flows through, before ending at an output layer thatproduces the neural network’s predictions. The network is trained to output useful predictions byidentifying patterns in a set of labeled training data, fed through the network while the outputs arecompared with the actual labels by an objective function. During training the network’s parameters–the strength of each neuron–is tuned until the patterns identified by the network result in good

10https://www.deeplearningbook.org/11The loose connection between artificial neural networks and neural networks in the brain is often mentioned, but

quite over-blown considering the complexity of biological neural networks. However, there is some interesting recentwork connecting neuroscience and artificial neural networks, indicating an increase in the cross-fertilization betweenthe two fields [43, 44, 45].

3

predictions for the training data. Once the patterns are learned, the network can be used to makepredictions on new, unseen data, i.e. generalize to new data.

It has long been known that ANNs are very flexible, able to model and solve complicatedproblems, but also that they are difficult and very computationally expensive to train.12 This haslowered their practical utility and led people to, until recently, focus on other machine learningmodels. But by now, artificial neural networks form one of the dominant methods in machinelearning, and the most intensively studied. This change is thanks to the growth of big data, powerfulprocessors for parallel computations (in particular, GPUs), some important tweaks to the algorithmsused to construct and train the networks, and the development of easy-to-use software frameworks.The surge of interest in ANNs leads to an incredible pace of developments, which also drives otherparts of machine learning with it.

The freely available books [41, 50] are two of the many excellent sources to learn more aboutartificial neural networks. We’ll only give a brief indication of how they are constructed and trained.The basic form of artificial neural networks13, the feedforward neural networks, are parametrizedmathematical functions y = f(x; θ) that maps an input x to an output y by feeding it through anumber of nonlinear transformations: f(x) = (fn ◦ · · · ◦ f1)(x). Here each component fk, called anetwork layer, consists of a simple linear transformation of the previous component’s output, fol-lowed by a nonlinear function: fk = σk(θTk fk−1). The nonlinear functions σk are typically sigmoidfunctions or ReLUs, as discussed below, and the θk are matrices of numbers, called the model’sweights. During the training phase, the network is fed training data and tasked with making pre-dictions at the output layer that match the known labels, each component of the network producingan expedient representation of its input. It has to learn how to best utilize the intermediate repre-sentations to form a complex hierarchical representation of the data, ending in correct predictionsat the output layer. Training a neural network means changing its weights to optimize the outputsof the network. This is done using an optimization algorithm, called gradient descent, on a functionmeasuring the correctness of the outputs, called a cost function or loss function. The basic ideasbehind training neural networks are simple: as training data is fed through the network, computethe gradient of the loss function with respect to every weight using the chain rule, and reduce theloss by changing these weights using gradient descent. But one quickly meets huge computationalchallenges when faced with complicated networks with thousands or millions of parameters and anexponential number of paths between the nodes and the network output. The techniques designedto overcome these challenges gets quite complicated. See [41, Chapter 8] and [51, Chapter 3 and 4]for detailed descriptions of the techniques and practical issues involved in training neural networks.

Artificial neural networks are often depicted as a network of nodes, as in Figure 1.14

12According to the famous universal approximation theorem for artificial neural networks [46, 47, 48, 49], ANNs aremathematically able to approximate any continuous function defined on compact subspaces of Rn, using finitely manyneurons. There are some restrictions on the activation functions, but these can be relaxed (allowing for ReLUs forexample) by restricting the function space. This is an existence theorem and successfully training a neural networkto approximate a given function is another matter entirely. However, the theorem does suggests that neural networksare reasonable to study and develop further, at least as an engineering endeavour aimed at realizing their theoreticalpowers.

13These are basic when compared to for example recurrent neural networks, whose architectures are more involved14As we shall see, modern architectures are often significantly more complicated than captured by the illustration

and equations above, with connections between non-consecutive layers, input fed in also at later layers, multipleoutputs, and much more.

4

Figure 1: Artificial neural networks are built from simple linear functions followed by nonlinearities. One of thesimplest class of neural network is the multilayer perceptron, or feedforward neural network, originating from the workof Rosenblatt in the 1950s [52]. It’s based on simple computational units, called neurons, organized in layers. Writing

i for the i-th layer and j for the j-th unit of that layer, the output of the j-th unit at the i-th layer is z(i)j =

(θ(i)j

)T

x.

Here x consists of the outputs from the previous layer after they are fed through a simple nonlinear function called anactivation function, typically a sigmoid function σ(z) = 1/(1 + e−z) or a rectified linear unit ReLU(z) = max(0, z) orsmall variations thereof. Each layer therefore computes a weighted sum of the all the outputs from the neurons in theprevious layers, followed by a nonlinearity. These are called the layer activations. Each layer activation is fed to thenext layer in the network, which performs the same calculation, until you reach the output layer, where the network’spredictions are produced. In the end, you obtain a hierarchical representation of the input data, where the earlierfeatures tend to be very general, getting increasingly specific towards the output. By feeding the network training data,propagated through the layers, the network is trained to perform useful tasks. A training data point (or, typically,a small batch of training points) is fed to the network, the outputs and local derivatives at each node are recorded,and the difference between the output prediction and the true label is measured by an objective function, such asmean absolute error (L1), mean squared error (L2), cross-entropy loss, or Dice loss, depending on the application.The derivative of the objective function with respect to the output is calculated, and used as a feedback signal. Thediscrepancy is propagated backwards through the network and all the weights are updated to reduce the error. This isachieved using backward propagation [53, 54, 55], which calculates the gradient of the objective function with respectto the weights in each node using the chain rule together with dynamic programming, and gradient descent [56], anoptimization algorithm tasked with improving the weights.

2.2. Deep learning

Traditionally, machine learning models are trained to perform useful tasks based on manuallydesigned features extracted from the raw data, or features learned by other simple machine learningmodels. In deep learning, the computers learn useful representations and features automatically,directly from the raw data, bypassing this manual and difficult step. By far the most common modelsin deep learning are various variants of artificial neural networks, but there are others. The maincommon characteristic of deep learning methods is their focus on feature learning : automaticallylearning representations of data. This is the primary difference between deep learning approaches

5

and more “classical” machine learning. Discovering features and performing a task is merged intoone problem, and therefore both improved during the same training process. See [39] and [41] forgeneral overviews of the field.

In medical imaging the interest in deep learning is mostly triggered by convolutional neuralnetworks (CNNs) [57]15, a powerful way to learn useful representations of images and other struc-tured data. Before it became possible to use CNNs efficiently, these features typically had to beengineered by hand, or created by less powerful machine learning models. Once it became possibleto use features learned directly from the data, many of the handcrafted image features were typi-cally left by the wayside as they turned out to be almost worthless compared to feature detectorsfound by CNNs.16 There are some strong preferences embedded in CNNs based on how they areconstructed, which helps us understand why they are so powerful. Let us therefore take a look atthe building blocks of CNNs.

Figure 2: Building blocks of a typical CNN. A slight modification of a figure in [59], courtesy of the author.

2.3. Building blocks of convolutional neural networks

When applying neural networks to images one can in principle use the simple feedforward neuralnetworks discussed above. However, having connections from all nodes of one layer to all nodes in thenext is extremely inefficient. A careful pruning of the connections based on domain knowledge, i.e.the structure of images, leads to much better performance. A CNN is a particular kind of artificialneural network aimed at preserving spatial relationships in the data, with very few connectionsbetween the layers. The input to a CNN is arranged in a grid structure and then fed throughlayers that preserve these relationships, each layer operation operating on a small region of theprevious layer (Fig. 2). CNNs are able to form highly efficient representation of the input data17,

15Interestingly, CNNs was applied in medical image analysis already in the early 90s, e.g. [58], but with limitedsuccess.

16However, combining hand-engineered features with CNN features is a very reasonable approach when low amountsof training data makes it difficult to learn good features automatically

17It’s interesting to compare this with the biological vision systems and their receptive fields of variable size (volumesin visual space) of neurons at different hierarchical levels

6

well-suited for image-oriented tasks. A CNN has multiple layers of convolutions and activations,often interspersed with pooling layers, and is trained using backpropagation and gradient descentas for standard artificial neural networks. See Section 2.1. In addition, CNNs typically have fully-connected layers at the end, which compute the final outputs.18

i) Convolutional layers: In the convolutional layers the activations from the previous layersare convolved with a set of small parameterized filters, frequently of size 3 × 3, collected in atensor W (j,i), where j is the filter number and i is the layer number. By having each filter sharethe exact same weights across the whole input domain, i.e. translational equivariance at eachlayer, one achieves a drastic reduction in the number of weights that need to be learned. Themotivation for this weight-sharing is that features appearing in one part of the image likelyalso appear in other parts. If you have a filter capable of detecting horizontal lines, say, thenit can be used to detect them wherever they appear. Applying all the convolutional filters atall locations of the input to a convolutional layer produces a tensor of feature maps.

ii) Activation layer: The feature maps from a convolutional layer are fed through nonlinearactivation functions. This makes it possible for the entire neural network to approximatealmost any nonlinear function [48, 49]19 The activation functions are generally the very simplerectified linear units, or ReLUs, defined as ReLU(z) = max(0, z), or variants like leaky ReLUsor parametric ReLUs.20 See [60, 61] for more information about these and other activationfunctions. Feeding the feature maps through an activation function produces new tensors,typically also called feature maps.

iii) Pooling: Each feature map produced by feeding the data through one or more convolutionallayer is then typically pooled in a pooling layer. Pooling operations take small grid regions asinput and produce single numbers for each region. The number is usually computed by usingthe max function (max-pooling) or the average function (average pooling). Since a small shiftof the input image results in small changes in the activation maps, the pooling layers gives theCNN some translational invariance.

A different way of getting the downsampling effect of pooling is to use convolutions with in-creased stride lengths. Removing the pooling layers simplifies the network architecture withoutnecessarily sacrificing performance [62].

Other common elements in many modern CNNs include

iv) Dropout regularization: A simple idea that gave a huge boost in the performance of CNNs.By averaging several models in an ensemble one tend to get better performance than whenusing single models. Dropout [63] is an averaging technique based on stochastic sampling ofneural networks.21 By randomly removing neurons during training one ends up using slightly

18Lately, so-called fully-convolution CNNs have become popular, in which average pooling across the whole inputafter the final activation layer replaces the fully-connected layers, significantly reducing the total number of weightsin the network.

19A neural network with only linear activations would only be able to perform linear approximation. Adding furtherlayers wouldn’t improve its expressiveness.

20Other options include exponential linear units (ELUs), and the now rarely used sigmoid or tanh activationfunctions.

21The idea of dropout is also used for other machine learning models, as in the DART technique for regression trees[64]

7

different networks for each batch of training data, and the weights of the trained network aretuned based on optimization of multiple variations of the network.22

iiv) Batch normalization: These layers are typically placed after activation layers, producingnormalized activation maps by subtracting the mean and dividing by the standard deviationfor each training batch. Including batch normalization layers forces the network to periodicallychange its activations to zero mean and unit standard deviation as the training batch hitsthese layers, which works as a regularizer for the network, speeds up training, and makes itless dependent on careful parameter initialization [67].

In the design of new and improved CNN architectures, these components are combined in increas-ingly complicated and interconnected ways, or even replaced by other more convenient operations.When architecting a CNN for a particular task there are multiple factors to consider, includingunderstanding the task to be solved and the requirements to be met, figuring out how to best feedthe data to the network, and optimally utilizing one’s budget for computation and memory con-sumption. In the early days of modern deep learning one tended to use very simple combinationsof the building blocks, as in Lenet [57] and AlexNet [1]. Later network architectures are much morecomplex, each generation building on ideas and insights from previous architectures, resulting inupdates to the state-of-the-art. Table 1 contains a short list of some famous CNN architectures,illustrating how the building blocks can be combined and how the field moves along.

Table 1: A far from exhaustive, non-chronological, list of CNN architectures and some high-level descriptions

AlexNet [1] The network that launched the current deep learning boom by winning the2012 ILSVRC competition by a huge margin. Notable features include theuse of RELUs, dropout regularization, splitting the computations on multipleGPUs, and using data augmentation during training. ZFNet [68], a relativelyminor modification of AlexNet, won the 2013 ILSVRC competition.

VGG [69] Popularized the idea of using smaller filter kernels and therefore deeper net-works (up to 19 layers for VGG19, compared to 7 for AlexNet and ZFNet),and training the deeper networks using pre-training on shallower versions.

GoogLeNet [70] Promoted the idea of stacking the layers in CNNs more creatively, as networksin networks, building on the idea of [71]. Inside a relatively standard archi-tecture (called the stem), GoogLeNet contains multiple inception modules, inwhich multiple different filter sizes are applied to the input and their resultsconcatenated. This multi-scale processing allows the module to extract fea-tures at different levels of detail simultaneously. GoogLeNet also popularizedthe idea of not using fully-connected layers at the end, but rather global av-erage pooling, significantly reducing the number of model parameters. It wonthe 2014 ILSVRC competition.

22In addition to increased model performance, dropout can also be used to produce robust uncertainty measuresin neural networks. By leaving dropout turned on also during inference one effectively performs variational inference[65, 59, 66]. This relates standard deep neural networks to Bayesian neural networks, synthesized in the field ofBayesian deep learning.

8

ResNet [72] Introduced skip connections, which makes it possible to train much deepernetworks. A 152 layer deep ResNet won the 2015 ILSVRC competition, andthe authors also successfully trained a version with 1001 layers. Having skipconnections in addition to the standard pathway gives the network the optionto simply copy the activations from layer to layer (more precisely, from ResNetblock to ResNet block), preserving information as data goes through the layers.Some features are best constructed in shallow networks, while others requiremore depth. The skip connections facilitate both at the same time, increasingthe network’s flexibility when fed input data. As the skip connections makethe network learn residuals, ResNets perform a kind of boosting.

Highway nets [73] Another way to increase depth based on gating units, an idea from Long ShortTerm Memory (LSTM) recurrent networks, enabling optimization of the skipconnections in the network. The gates can be trained to find useful combina-tions of the identity function (as in ResNets) and the standard nonlinearitythrough which to feed its input.

DenseNet [74] Builds on the ideas of ResNet, but instead of adding the activations producedby one layer to later layers, they are simply concatenated together. The origi-nal inputs in addition to the activations from previous layers are therefore keptat each layer (again, more precisely, between blocks of layers), preserving somekind of global state. This encourages feature reuse and lowers the number ofparameters for a given depth. DenseNets are therefore particularly well-suitedfor smaller data sets (outperforming others on e.g. Cifar-10 and Cifar-100).

ResNext [75] Builds on ResNet and GoogLeNet by using inception modules between skipconnections.

SENets [76] Squeeze-and-Excitation Networks, which won the ILSVRC 2017 competition,builds on ResNext but adds trainable parameters that the network can use toweigh each feature map, where earlier networks simply added them up. TheseSE-blocks allows the network to model the channel and spatial informationseparately, increasing the model capacity. SE-blocks can easily be added toany CNN model, with negligible increase in computational costs.

NASNet [77] A CNN architecture designed by a neural network, beating all the previoushuman-designed networks at the ILSVRC competition. It was created usingAutoML23, Google Brain’s reinforcement learning approach to architecturedesign [78]. A controller network (a recurrent neural network) proposes archi-tectures aimed to perform at a specific level for a particular task, and by trialand error learns to propose better and better models. NASNet was based onCifar-10, and has relatively modest computational demands, but still outper-formed the previous state-of-the-art on ILSVRC data.

23https://cloud.google.com/automl

9

YOLO [79] Introduced a new, simplified way to do simultaneous object detection and clas-sification in images. It uses a single CNN operating directly on the image andoutputting bounding boxes and class probabilities. It incorporates several el-ements from the above networks, including inception modules and pretraininga smaller version of the network. It’s fast enough to enable real-time pro-cessing24. YOLO makes it easy to trade accuracy for speed by reducing themodel size. YOLOv3-tiny was able to process images at over 200 frames persecond on a standard benchmark data set, while still producing reasonablepredictions.

GANs [80] A generative adversarial network consists of two neural networks pitted againsteach other. The generative network G is tasked with creating samples that thediscriminative network D is supposed to classify as coming from the generativenetwork or the training data. The networks are trained simultaneously, whereG aims to maximize the probability that D makes a mistake while D aims forhigh classification accuracy.

Siamese nets [81] An old idea (e.g. [82]) that’s recently been shown to enable one-shot learning,i.e. learning from a single example. A siamese network consists of two identicalneural networks, both the architecture and the weights, attached at the end.They are trained together to differentiate pairs of inputs. Once trained, thefeatures of the networks can be used to perform one-shot learning withoutretraining.

U-net [83] A very popular and successful network for segmentation in 2D images. Whenfed an input image, it is first downsampled through a “traditional” CNN, be-fore being upsampled using transpose convolutions until it reaches its originalsize. In addition, based on the ideas of ResNet, there are skip connectionsthat concatenates features from the downsampling to the upsampling paths.It is a fully-convolutional network, using the ideas first introduced in [84].

V-net [85] A three-dimensional version of U-net with volumetric convolutions and skip-connections as in ResNet.

These neural networks are typically implemented in one or more of a small number of softwareframeworks that dominates machine learning research, all built on top of NVIDIA’s CUDA plat-form and the cuDNN library. Today’s deep learning methods are almost exclusively implementedin either TensorFlow, a framework originating from Google Research, Keras, a deep learning li-brary originally built by Francois Chollet and recently incorporated in TensorFlow, or Pytorch, aframework associated with Facebook Research. There are very few exceptions (YOLO built usingthe Darknet framework [86] is one of the rare ones). All the main frameworks are open source andunder active development.

3. Deep learning, medical imaging and MRI

Deep learning methods are increasingly used to improve clinical practice, and the list of examplesis long, growing daily. We will not attempt a comprehensive overview of deep learning in medicalimaging, but merely sketch some of the landscape before going into a more systematic expositionof deep learning in MRI.

24You can watch YOLO in action here https://youtu.be/VOC3huqHrss

10

Convolutional neural networks can be used for efficiency improvement in radiology practicesthrough protocol determination based on short-text classification [87]. They can also be used toreduce the gadolinium dose in contrast-enhanced brain MRI by an order of magnitude [88] withoutsignificant reduction in image quality. Deep learning is applied in radiotherapy [89], in PET-MRIattenuation correction [90, 91], in radiomics [92, 93] (see [94] for a review of radiomics relatedto radiooncology and medical physics), and for theranostics in neurosurgical imaging, combiningconfocal laser endomicroscopy with deep learning models for automatic detection of intraoperativeCLE images on-the-fly [95].

Another important application area is advanced deformable image registration, enabling quan-titative analysis across different physical imaging modalities and across time.25. For example elasticregistration between 3D MRI and transrectal ultrasound for guiding targeted prostate biopsy [96];deformable registration for brain MRI where a “cue-aware deep regression network” learns froma given set of training images the displacement vector associated with a pair of reference-subjectpatches [97]; fast deformable image registration of brain MR image pairs by patch-wise predictionof the Large Deformation Diffeomorphic Metric Mapping model [98]26; unsupervised convolutionalneural network-based algorithm for deformable image registration of cone-beam CT to CT usinga deep convolutional inverse graphics network [99]; deep learning-based 2D/3D registration frame-work for registration of preoperative 3D data and intraoperative 2D X-ray images in image-guidedtherapy [100]; real-time prostate segmentation during targeted prostate biopsy, utilizing temporalinformation in the series of ultrasound images [101].

This is just a tiny sliver of the many applications of deep learning to central problems in medicalimaging. There are several thorough reviews and overviews of the field to consult for more informa-tion, across modalities and organs, and with different points of view and level of technical details. Forexample the comprehensive review [102]27, covering both medicine and biology and spanning fromimaging applications in healthcare to protein-protein interaction and uncertainty quantification; keyconcepts of deep learning for clinical radiologists [103, 104, 105, 106, 107, 108, 109, 110, 111, 112],including radiomics and imaging genomics (radiogenomics) [113], and toolkits and libraries for deeplearning [114]; deep learning in neuroimaging and neuroradiology [115]; brain segmentation [116];stroke imaging [117, 118]; neuropsychiatric disorders [119]; breast cancer [120, 121]; chest imaging[122]; imaging in oncology [123, 124, 125]; medical ultrasound [126, 127]; and more technical sur-veys of deep learning in medical image analysis [42, 128, 129, 130]. Finally, for those who like tobe hands-on, there are many instructive introductory deep learning tutorials available online. Forexample [131], with accompanying code available at https://github.com/paras42/Hello_World_Deep_Learning, where you’ll be guided through the construction of a system that can differentiatea chest X-ray from an abdominal X-ray using the Keras/TensorFlow framework through a JupyterNotebook. Other nice tutorials are http://bit.ly/adltktutorial, based on the Deep LearningToolkit (DLTK) [132], and https://github.com/usuyama/pydata-medical-image, based on theMicrosoft Cognitive Toolkit (CNTK).

Let’s now turn to the field of MRI, in which deep learning has seen applications at each stepof entire workflows. From acquisition to image retrieval, from segmentation to disease prediction.We divide this into two parts: (i) the signal processing chain close to the physics of MRI, includingimage restoration and multimodal image registration (Fig. 3), and (ii) the use of deep learning in

25e.g. test-retest examinations, or motion correction in dynamic imaging26available at https://github.com/rkwitt/quicksilver27A continuous collaborative manuscript (https://greenelab.github.io/deep-review) with >500 references.

11

MR image segmentation, disease detection, disease prediction and systems based on images andtext data (reports), addressing a few selected organs such as the brain, the kidney, the prostate andthe spine (Fig. 4).

3.1. From image acquisition to image registration

Deep learning in MRI has typically been focused on segmentation and classification of recon-structed magnitude images. Its penetration into the lower levels of MRI measurement techniquesis more recent, but already impressive. From MR image acquisition and signal processing in MRfingerprinting, to denoising and super-resolution, and into image synthesis.

Magnitude

PhaseFFT-1

w

y

x

RF

z

Re

Im

k-space

IMAGE ACQUISITION IMAGE RECONSTRUCTION IMAGE RESTORATION IMAGE REGISTRATION

sMRI

dMRI

fMRI

MultiparametricMRI

Figure 3: Deep learning in the MR signal processing chain, from image acquisition (in complex-valued k-space) andimage reconstruction, to image restoration (e.g. denoising) and image registration. The rightmost column illustratescoregistration of multimodal brain MRI. sMRI = structural 3D T1-weighted MRI, dMRI = diffusion weighted MRI(stack of slices in blue superimposed on sMRI), fMRI = functional BOLD MRI (in red).

3.1.1. Data acquisition and image reconstruction

Research on CNN and RNN-based image reconstruction methods is rapidly increasing, pioneeredby Yang et al. [133] at NIPS 2016 and Wang et al. [134] at ISBI 2016. Recent applications ad-dresses e.g. convolutional recurrent neural networks for dynamic MR image reconstruction [135],reconstructing good quality cardiac MR images from highly undersampled complex-valued k-spacedata by learning spatio-temporal dependencies, outperforming 3D CNN approaches and compressedsensing-based dynamic MRI reconstruction algorithms in computational complexity, reconstructionaccuracy and speed for different undersampling rates. Schlemper et.al. [136] created a deep cascadeof concatenated CNNs for dynamic MR image reconstruction, making use of data augmentation,both rigid and elastic deformations, to increase the variation of the examples seen by the networkand reduce overfitting28. Using variational networks for single-shot fast spin-echo MRI with variabledensity sampling, Chen et.al. [137] enabled real-time (200 ms per section) image reconstruction,outperforming conventional parallel imaging and compressed sensing reconstruction. In [138], theauthors explored the potential for transfer learning (pretrained models) and assessed the gener-alization of learned image reconstruction regarding image contrast, SNR, sampling pattern andimage content, using a variational network and true measurement k-space data from patient kneeMRI recordings and synthetic k-space data generated from images in the Berkeley SegmentationData Set and Benchmarks. Employing least-squares generative adversarial networks (GANs) that

28Code available at https://github.com/js3611/Deep-MRI-Reconstruction

12

learns texture details and suppresses high-frequency noise, [139] created a novel compressed sensingframework that can produce diagnostic quality reconstructions “on the fly” (30 ms)29. A unifiedframework for image reconstruction [140], called automated transform by manifold approximation(AUTOMAP) consisting of a feedforward deep neural network with fully connected layers followedby a sparse convolutional autoencoder, formulate image reconstruction generically as a data-drivensupervised learning task that generates a mapping between the sensor and the image domain basedon an appropriate collection of training data (e.g. MRI examinations collected from the HumanConnectome Project, transformed to the k-space sensor domain).

There are also other approaches and reports on deep learning in MR image reconstruction, e.g.[141, 142, 143, 144], a fundamental field rapidly progressing.

3.1.2. Quantitative parameters - QSM and MR fingerprinting

Another area that is developing within deep learning for MRI is the estimation of quantitativetissue parameters from recorded complex-valued data. For example within quantitative susceptibilitymapping, and in the exciting field of magnetic resonance fingerprinting.

Quantitative susceptibility mapping (QSM) is a growing field of research in MRI, aiming tononinvasively estimate the magnetic susceptibility of biological tissue [145, 146]. The technique isbased on solving the difficult, ill-posed inverse problem of determining the magnetic susceptibilityfrom local magnetic fields. Recently Yoon et al. [147] constructed a three-dimensional CNN, namedQSMnet and based on the U-Net architecture, able to generate high quality susceptibility sourcemaps from single orientation data. The authors generated training data by using the gold-standardfor QSM: the so-called COSMOS method [148]. The data was based on 60 scans from 12 healthyvolunteers. The resulting model both simplified and improved the state-of-the-art for QSM. Ras-mussen and coworkers [149] took a different approach. They also used a U-Net-based convolutionalneural network to perform field-to-source inversion, called DeepQSM, but it was trained on syn-thetically generated data containing simple geometric shapes such as cubes, rectangles and spheres.After training their model on synthetic data it was able to generalize to real-world clinical brainMRI data, computing susceptibility maps within seconds end-to-end. The authors conclude thattheir method, combined with fast imaging sequences, could make QSM feasible in standard clinicalpractice.

Magnetic resonance fingerprinting (MRF) was introduced a little more than five years ago [150],and has been called “a promising new approach to obtain standardized imaging biomarkers fromMRI” by the European Society of Radiology [151]. It uses a pseudo-randomized acquisition thatcauses the signals from different tissues to have a unique signal evolution (“fingerprint”) that is afunction of the multiple material properties being investigated. Mapping the signals back to knowntissue parameters (T1, T2 and proton density) is then a rather difficult inverse problem. MRF isclosely related to the idea of compressed sensing [152] in MRI [153] in that MRF undersamplesdata in k-space producing aliasing artifacts in the reconstructed images that can be suppressed bycompressed sensing.30 It can be regarded as a quantitative multiparametric MRI analysis, and withrecent acquisition schemes using a single-shot spiral trajectory with undersampling, whole-braincoverage of T1, T2 and proton density maps can be acquired at 1.2 × 1.2 × 3 mm3 voxel resolution

29In their GAN setting, a generator network is used to map undersampled data to a realistic-looking image withhigh measurement fidelity, while a discriminator network is trained jointly to score the quality of the reconstructedimage.

30See [154, 155, 156, 157, 158] for recent perspectives and developments connecting deep learning-based reconstruc-tion methods to the more general research field of inverse problems.

13

in less than 5 min [159].The processing of MRF after acquisition usually involves using various pattern recognition algo-

rithms that try to match the fingerprints to a predefined dictionary of predicted signal evolutions31,created using the Bloch equations [150, 164].

Recently, deep learning methodology has been applied to MR fingerprinting. Cohen et al.[165] reformulated the MRF reconstruction problem as learning an optimal function that mapsthe recorded signal magnitudes to the corresponding tissue parameter values, trained on a sparseset of dictionary entries. To achieve this they fed voxel-wise MRI data acquired with an MRFsequence (MRF-EPI, 25 frames in ∼3 s; or MRF-FISP, 600 frames in ∼7.5 s) to a four-layer neuralnetwork consisting of two hidden layers with 300 × 300 fully connected nodes and two nodes inthe output layer, considering only T1 and T2 parametric maps. The network, called MRF DeepRecOnstruction NEtwork (DRONE), was trained by an adaptive moment estimation stochasticgradient descent algorithm with a mean squared error loss function. Their dictionary consisted of∼70000 entries (product of discretized T1 and T2 values) and training the network to convergencewith this dictionary (∼10 MB for MRF-EPI and ∼300 MB for MRF-FISP) required 10 to 70 minusing an NVIDIA K80 GPU with 2 GB memory. They found their reconstruction time (10 to 70 msper slice) to be 300 to 5000 times faster than conventional dictionary-matching techniques, usingboth well-characterized calibrated ISMRM/NIST phantoms and in vivo human brains.

A similar deep learning approach to predict quantitative parameter values (T1 and T2) fromMRF time series was taken by Hoppe et al. [166]. In their experiments they used 2D MRF-FISP data with variable TR (12-15 ms), flip angles (5◦-74◦) and 3000 repetitions, recorded on aMAGNETOM 3T Skyra. A high resolution dictionary was simulated to generate a large collectionof training and testing data, using tissues T1 and T2 relaxation time ranges as present in normalbrain at 3T (e.g. [167]) resulting in ∼ 1.2 × 105 time series. In contrast to [165], their deep neuralnetwork architecture was inspired from the domain of speech recognition due to the similarity ofthe two tasks. The architecture with the smallest average error for validation data was a standardconvolutional neural network consisting of an input layer of 3000 nodes (number of samples inthe recorded time series), four hidden layers, and an output layers with two nodes (T1 and T2).Matching one time series was about 100 times faster than the conventional [150] matching methodand with very small mean absolute deviations from ground truth values.

In the same context, Fang et al. [168] used a deep learning method to extract tissue propertiesfrom highly undersampled 2D MRF-FISP data in brain imaging, where 2300 time points wereacquired from each measurement and each time point consisted of data from one spiral readoutonly. The real and imaginary parts of the complex signal were separated into two channels. Theyused MRF signal from a patch of 32 × 32 pixels to incorporate correlated information betweenneighboring pixels. In their work they designed a standard three-layer CNN with T1 and T2 asoutput.

Virtue et.al. [169] investigated a different approach to MRF. By generating 100.000 syntheticMRI signals using a Bloch equation simulator they were able to train feedforward deep neural net-works to map new MRI signals to the tissue parameters directly, producing approximate solutionsto the inverse mapping problem of MRF. In their work they designed a new complex activationfunction, the complex cardioid, that was used to construct a complex-valued feedforward neuralnetwork. This three-layer network outperformed both the standard MRF techniques based on dic-

31A dictionary of time series for every possible combination of parameters like (discretized) T1 and T2 relaxationtimes, spin-density (M0), B0, off-resonance (∆f), and also voxel-wise cerebral blood volume (CBV), mean vesselradius (R), blood oxygen saturation (SO2) and T∗

2 [160, 161, 162], and more, e.g. MFR-ASL [163].

14

tionary matching, and also the analogous real neural network operating on the real and imaginarycomponents separately. This suggested that complex-valued networks are better suited at uncover-ing information in complex data.32

3.1.3. Image restoration (denoising, artifact detection)

Estimation of noise and image denoising in MRI has been an important field of research for manyyears [172, 173], employing a plethora of methods. For example Bayesian Markov random fieldmodels [174], rough set theory [175], higher-order singular value decomposition [176], wavelets [177],independent component analysis [178], or higher order PDEs [179].

Recently, deep learning approaches have been introduced to denoising. In their work on learningimplicit brain MRI manifolds using deep neural networks, Bermudez et al. [180] implemented anautoencoder with skip connections for image denoising, testing their approach with adding variouslevels of Gaussian noise to more than 500 T1-weighted brain MR images from healthy controls in theBaltimore Longitudinal Study of Aging. Their autoencoder network outperformed the current FSLSUSAN denoising software according to peak signal-to-noise ratios. Benou et al. [181] addressedspatio-temporal denoising of dynamic contrast-enhanced MRI of the brain with bolus injection ofcontrast agent (CA), proposing a novel approach using ensembles of deep neural networks for noisereduction. Each DNN was trained on a different range of SNRs and types of CA concentrationtime curves (denoted “pathology experts”, “healthy experts”, “vessel experts”) to generate a re-construction hypothesis from noisy input by using a classification DNN to select the most likelyhypothesis and provide a “clean output” curve. Training data was generated synthetically using athree-parameter Tofts pharmacokinetic (PK) model and noise realizations. To improve this model,accounting for spatial dependencies of PK pharmacokinetics, they used concatenated noisy timecurves from first-order neighbourhood pixels in their expert DNNs and ensemble hypothesis DNN,collecting neighboring reconstructions before a boosting procedure produced the final clean out-put for the pixel of interest. They tested their trained ensemble model on 33 patients from twodifferent DCE-MRI databases with either stroke or recurrent glioblastoma (RIDER NEURO33),acquired at different sites, with different imaging protocols, and with different scanner vendors andfield strengths. The qualitative and quantitative (MSE) denoising results were better than spatio-temporal Beltrami, moving average, the dynamic Non Local Means method [182], and stackeddenoising autoencoders [183]. The run-time comparisons were also in favor of the proposed sDNN.

In this context of DCE-MRI, it’s tempting to speculate whether deep neural network approachescould be used for direct estimation of tracer-kinetic parameter maps from highly undersampled(k, t)-space data in dynamic recordings [184, 185], a powerful way to by-pass 4D DCE-MRI recon-struction altogether and map sensor data directly to spatially resolved pharmacokinetic parameters,e.g. Ktrans, vp, ve in the extended Tofts model or parameters in other classic models [186]. A relatedapproach in the domain of diffusion MRI, by-passing the model-fitting steps and computing voxel-wise scalar tissue properties (e.g. radial kurtosis, fiber orientation dispersion index) directly fromthe subsampled DWIs was taken by Golkov et al. [187] in their proposed “q-space deep learning”family of methods.

Deep learning methods has also been applied to MR artifact detection, e.g. poor quality spectrain MRSI [188]; detection and removal of ghosting artifacts in MR spectroscopy [189]; and automatedreference-free detection of patient motion artifacts in MRI [190].

32Complex-valued deep learning is also getting some attention in a broader community of researchers, and has beenshown to lead to improved models. See e.g. [170, 171] and the references therein.

33https://wiki.cancerimagingarchive.net/display/Public/RIDER+NEURO+MRI

15

3.1.4. Image super-resolution

Image super-resolution, reconstructing a higher-resolution image or image sequence from the ob-served low-resolution image [191], is an exciting application of deep learning methods34.

Super-resolution for MRI have been around for almost 10 years [192, 193] and can be used toimprove the trade-off between resolution, SNR, and acquisition time [194], generate 7T-like MRimages on 3T MRI scanners [195], or obtain super-resolution T1 maps from a set of low resolutionT1 weighted images [196]. Recently deep learning approaches has been introduced, e.g. generatingsuper-resolution single (no reference information) and multi-contrast (applying a high-resolutionimage of another modality as reference) brain MR images using CNNs [197]; constructing super-resolution brain MRI by a CNN stacked by multi-scale fusion units [198]; and super-resolutionmusculoskeletal MRI (“DeepResolve”) [199]. In DeepResolve thin (0.7 mm) slices in knee images(DESS) from 124 patients included in the Osteoarthritis Initiative were used for training and 17patients for testing, with a 10s inference time per 3D (344 × 344 × 160) volume. The resulting im-ages were evaluated both quantitatively (MSE, PSNR, and the perceptual window-based structuralsimilarity SSIM35 index) and qualitatively by expert radiologists.

3.1.5. Image synthesis

Image synthesis in MRI have traditionally been seen as a method to derive new parametric imagesor new tissue contrast from a collection of MR acquisition performed at the same imaging session,i.e. “an intensity transformation applied to a given set of input images to generate a new image witha specific tissue contrast” [200]. Another avenue of MRI synthesis is related to quantitative imagingand the development and use of physical phantoms, imaging calibration/standard test objects withspecific material properties. This is done in order to assess the performance of an MRI scanner orto assess imaging biomarkers reliably with application-specific phantoms such as a structural brainimaging phantom, DCE-MRI perfusion phantom, diffusion phantom, flow phantom, breast phantomor a proton-density fat fraction phantom [201]. The in silico modeling of MR images with certainunderlying properties, e.g. [202, 203], or model-based generation of large databases of (cardiac)images from real healthy cases [204] is also part of this endeavour. In this context, deep learningapproaches have accelerated research and the amount of costly training data.

The last couple of years have seen impressive results for photo-realistic image synthesis using deeplearning techniques, especially generative adversarial networks (GANs, introduced by Goodfellow etal. in 2014 [80]), e.g. [205, 206, 207]. These can also be used for biological image synthesis [208, 209]and text-to-image synthesis [210, 211, 212].36 Recently, a group of researchers from NVIDIA,MGH & BWH Center for Clinical Data Science in Boston, and the Mayo Clinic in Rochester[213] designed a clever approach to generate synthetic abnormal MRI images with brain tumors bytraining a GAN based on pix2pix37 using two publicly available data sets of brain MRI (ADNIand the BRATS’15 Challenge, and later also the Ischemic Stroke Lesion Segmentation ISLES’2018Challenge). This approach is highly interesting as medical imaging datasets are often imbalanced,with few pathological findings, limiting the training of deep learning models. Such generativemodels for image synthesis serve as a form of data augmentation, and also as an anonymizationtool. The authors achieved comparable tumor segmentation results when trained on the synthetic

34See http://course.fast.ai/lessons/lesson14.html for an instructive introduction to super-resolution35http://www.cns.nyu.edu/~lcv/ssim36See here https://github.com/xinario/awesome-gan-for-medical-imaging for a list of interesting applications

of GAN in medical imaging37https://phillipi.github.io/pix2pix

16

data rather than on real patient data. A related approach to brain tumor segmentation using coarse-to-fine GANs was taken by Mok & Chung [214]. Guibas et al. [215] used a two-stage pipeline forgenerating synthetic medical images from a pair of GANs, addressing retinal fundus images, andprovided an online repository (SynthMed) for synthetic medical images. Kitchen & Seah [216] usedGANs to synthetize realistic prostate lesions in T2, ADC, Ktrans resembling the SPIE-AAPM-NCIProstateX Challenge 201638 training data.

Other applications are unsupervised synthesis of T1-weighted brain MRI using a GAN [180];image synthesis with context-aware GANs [217]; synthesis of patient-specific transmission imagefor PET attenuation correction in PET/MR imaging of the brain using a CNN [218]; pseudo-CTsynthesis for pelvis PET/MR attenuation correction using a Dixon-VIBE Deep Learning (DIVIDE)network [219]; image synthesis with GANs for tissue recognition [220]; synthetic data augmentationusing a GAN for improved liver lesion classification [221]; and deep MR to CT synthesis usingunpaired data [222].

3.1.6. Image registration

Image registration39 is an increasingly important field within MR image processing and analysis asmore and more complementary and multiparametric tissue information are collected in space andtime within shorter acquisition times, at higher spatial (and temporal) resolutions, often longitudi-nally, and across patient groups, larger cohorts, or atlases. Traditionally one has divided the tasksof image registration into dichotomies: intra vs. inter-modality, intra vs. inter-subject, rigid vs. de-formable, geometry-based vs. intensity-based, and prospective vs. retrospective image registration.Mathematically, registration is a challenging mix of geometry (spatial transformations), analysis(similarity measures), optimization strategies, and numerical schemes. In prospective motion cor-rection, real-time MR physics is also an important part of the picture [224, 225]. A wide range ofmethodological approaches have been developed and tested for various organs and applications40

[229, 230, 231, 232, 233, 234, 235, 236, 237, 238], including “previous generation” artificial neuralnetworks [239].

Recently, deep learning methods have been applied to image registration in order to improveaccuracy and speed (e.g. Section 3.4 in [42]). For example: deformable image registration [240,98]; model-to-image registration [241, 242]; MRI-based attenuation correction for PET [243, 244];PET/MRI dose calculation [245]; unsupervised end-to-end learning for deformable registration of2D CT/MR images [246]; an unsupervised learning model for deformable, pairwise 3D medicalimage registration by Balakrishnan et al. [247]41; and a deep learning framework for unsupervisedaffine and deformable image registration [248].

3.2. From image segmentation to diagnosis and prediction

We leave the lower-level applications of deep learning in MRI to consider higher-level (down-stream) applications such as fast and accurate image segmentation, disease prediction in selectedorgans (brain, kidney, prostate, and spine) and content-based image retrieval, typically applied

38https://www.aapm.org/GrandChallenge/PROSTATEx-239Image registration can be defined as “the determination of a one-to-one mapping between the coordinates in one

space and those in another, such that points in the two spaces that correspond to the same anatomical point aremapped to each other” (C.R Maurer [223], 1993).

40and different hardware e.g. GPUs [226, 227, 228] as image registration is often computationally time consuming.41with code available at https://github.com/voxelmorph/voxelmorph

17

to reconstructed magnitude images. We have chosen to focus our overview on deep learning ap-plications close to the MR physics and will be brief in the present section, even if the followingapplications are very interesting and clinically important.

BRAIN KIDNEY PROSTATE SPINE

Figure 4: Deep learning for MR image analysis in selected organs, partly from ongoing work at MMIV.

3.2.1. Image segmentation

Image segmentation, the holy grail of quantitative image analysis, is the process of partitioningan image into multiple regions that share similar attributes, enabling localization and quantifi-cation.42 It has an almost 50 years long history, and has become the biggest target for deeplearning approaches in medical imaging. The multispectral tissue classification report by Van-nier et al. in 1985 [249], using statistical pattern recognition techniques (and satellite image pro-cessing software from NASA), represented one of the most seminal works leading up to today’smachine learning in medical imaging segmentation. In this early era, we also had the opportu-nity to contribute with supervised and unsupervised machine learning approaches for MR imagesegmentation and tissue classification [250, 251, 252, 253]. An impressive range of segmentationmethods and approaches have been reported (especially for brain segmentation) and reviewed, e.g.[254, 255, 256, 257, 258, 259, 260, 261, 262]. MR image segmentation using deep learning ap-proaches, typically CNNs, are now penetrating the whole field of applications. For example acuteischemic lesion segmentation in DWI [263]; brain tumor segmentation [264]; segmentation of thestriatum [265]; segmentation of organs-at-risks in head and neck CT images [266]; and fully auto-mated segmentation of polycystic kidneys [267]; deformable segmentation of the prostate [268]; andspine segmentation with 3D multiscale CNNs [269].

See [42] and [102] for more comprehensive lists.

3.2.2. Diagnosis and prediction

A presumably complete list of papers up to 2017 using deep learning techniques for brain imageanalysis is provided as Table 1 in Litjens at al. [42]. In the following we add some more recentwork on organ-specific deep learning using MRI, restricting ourselves to brain, kidney, prostate andspine.

Table 2: A short list of deep learning applications per organ, task, reference and description.

BRAIN

42Segmentation is also crucial for functional imaging, enabling tissue physiology quantification with preservation ofanatomical specificity.

18

Brain extraction [270] A 3D CNN for skull stripping

Functional connectomes [271] Transfer learning approach to enhance deep neural network classifica-tion of brain functional connectomes

[272] Multisite diagnostic classification of schizophrenia using discriminantdeep learning with functional connectivity MRI

Structural connectomes [273] A convolutional neural network-based approach (https://github.com/MIC-DKFZ/TractSeg) that directly segments tracts in the field of fiberorientation distribution function (fODF) peaks without using tractog-raphy, image registration or parcellation. Tested on 105 subjects fromthe Human Connectome Project

Brain age [274] Chronological age prediction from raw brain T1-MRI data, also testingthe heritability of brain-predicted age using a sample of 62 monozygoticand dizygotic twins

Alzheimer’s disease [275] Landmark-based deep multi-instance learning evaluated on 1526 sub-jects from three public datasets (ADNI-1, ADNI-2, MIRIAD)

[276] Identify different stages of AD

[277] Multimodal and multiscale deep neural networks for the early diagnosisof AD using structural MR and FDG-PET images

Vascular lesions [278] Evaluation of a deep learning approach for the segmentation of braintissues and white matter hyperintensities of presumed vascular originin MRI

Identification of MRIcontrast

[279] Using deep learning algorithms to automatically identify the brain MRIcontrast, with implications for managing large databases

Meningioma [280] Fully automated detection and segmentation of meningiomas usingdeep learning on routine multiparametric MRI

Glioma [281] Glioblastoma segmentation using heterogeneous MRI data from clinicalroutine

[282] Deep learning for segmentation of brain tumors and impact of cross-institutional training and testing

[283] Automatic semantic segmentation of brain gliomas from MRI using adeep cascaded neural network

[284] AdaptAhead optimization algorithm for learning deep CNN applied toMRI segmentation of glioblastomas (BRATS)

Multiple sclerosis [285] Deep learning of joint myelin and T1w MRI features in normal-appearing brain tissue to distinguish between multiple sclerosis patientsand healthy controls

KIDNEY

Abdominal organs [286] CNNs to improve abdominal organ segmentation, including left kidney,right kidney, liver, spleen, and stomach in T2-weighted MR images

19

Cyst segmentation [267] An artificial multi-observer deep neural network for fully automatedsegmentation of polycystic kidneys

Renal transplant [287] A deep-learning-based classifier with stacked non-negative constrainedautoencoders to distinguish between rejected and non-rejected renaltransplants in DWI recordings

PROSTATE

Cancer (PCa) [288] Proposed a method for end-to-end prostate segmentation by integratingholistically (image-to-image) nested edge detection with fully convolu-tional networks. their nested networks automatically learn a hierar-chical representation that can improve prostate boundary detection.Obtained very good results (Dice coefficient, 5-fold cross validation) onMRI scans from 250 patients

[289] Computer-aided diagnosis with a CNN, deciding ‘cancer’ ‘no cancer’trained on data from 301 patients with a prostate-specific antigen levelof < 20 ng/mL who underwent MRI and extended systematic prostatebiopsy with or without MRI-targeted biopsy

[290] Automatic approach based on deep CNN, inspired from VGG, to clas-sify PCa and noncancerous tissues with multiparametric MRI usingdata from the PROSTATEx database

[291] Deep CNN and a non-deep learning using feature detection (the scale-invariant feature transform and the bag-of-words model, a representa-tive method for image recognition and analysis) were used to distinguishpathologically confirmed PCa patients from prostate benign conditionspatients with prostatitis or prostate benign hyperplasia in a collectionof 172 patients with more than 2500 morphologic 2D T2-w MR images

[292] Designed a system which can concurrently identify the presence of PCain an image and localize lesions based on deep CNN features (co-trainedCNNs consisting of two parallel convolutional networks for ADC andT2-w images respectively) and a single-stage SVM classifier for au-tomated detection of PCa in multiparametric MRI. Evaluated on adataset of 160 patients

[293] Designed and tested multimodel CNNs, using clinical data from 364patients with a total of 463 PCa lesions and 450 identified noncancer-ous image patches. Carefully investigated three critical factors whichcould greatly affect the performance of their multimodal CNNs buthad not been carefully studied previously: (1) Given limited trainingdata, how can these be augmented in sufficient numbers and variety forfine-tuning deep CNN networks for PCa diagnosis? (2) How can mul-timodal mp-MRI information be effectively combined in CNNs? (3)What is the impact of different CNN architectures on the accuracy ofPCa diagnosis?

SPINE

Vertebrae labeling [294] Designed a CNN for detection and labeling of vertebrae in MR imageswith clinical annotations as training data

Intervertebral disc local-ization

[269] 3D multi-scale fully connected CNNs with random modality voxeldropout learning for intervertebral disc localization and segmentationfrom multi-modality MR images

20

Disc-level labeling,spinal stenosis grading

[295] CNN model denoted DeepSPINE, having a U-Net architecture com-bined with a spine-curve fitting method for automated lumbar verte-bral segmentation, disc-level designation, and spinal stenosis gradingwith a natural language processing scheme

Lumbal neural forminalstenosis (LNFS)

[296] Addressed the challenge of automated pathogenesis-based diagnosis, si-multaneously localizing and grading multiple spinal structures (neuralforamina, vertebrae, intervertebral discs) for diagnosing LNFS and dis-cover pathogenic factors. Proposed a deep multiscale multitask learningnetwork integrating a multiscale multi-output learning and a multi-task regression learning into a fully convolutional network where (i) aDMML-Net merges semantic representations to reinforce the salienceof numerous target organs (ii) a DMML-Net extends multiscale convo-lutional layers as multiple output layers to boost the scale-invariancefor various organs, and (iii) a DMML-Net joins the multitask regressionmodule and the multitask loss module to combine the mutual benefitbetween tasks

Spondylitis vs tuberculo-sis

[297] CNN model for differentiating between tuberculous and pyogenicspondylitis in MR images. Compared their CNN performance withthat of three skilled radiologists using spine MRIs from 80 patients

Metastasis [291] A multi-resolution approach for spinal metastasis detection using deepSiamese neural networks comprising three identical subnetworks formulti-resolution analysis and detection. Detection performance wasevaluated on a set of 26 cases using a free-response receiver operat-ing characteristic analysis (observer is free to mark and rate as manysuspicious regions as are considered clinically reportable)

3.3. Content-based image retrieval

The objective of content-based image retrieval (CBIR) in radiology is to provide medical casessimilar to a given image in order to assist radiologists in the decision-making process. It typicallyinvolves large case databases, clever image representations and lesion annotations, and algorithmsthat are able to quickly and reliably match and retrieve the most similar images and their anno-tations in the case database. CBIR has been an active area of research in medical imaging formany years, addressing a wide range of applications, imaging modalities, organs, and method-ological approaches, e.g. [298, 299, 300, 301, 302, 303, 304], and at a larger scale outside themedical field using deep learning techniques, e.g. at Microsoft, Apple, Facebook, and Google(reverse image search43), and others. See e.g. [305, 306, 307, 308, 309] and the code reposito-ries https://github.com/topics/image-retrieval. One of the first application of deep learning forCBIR in the medical domain came in 2015 when Sklan et al. [310] trained a CNN to perform CBIRwith more than one million random MR and CT images, with disappointing results (true positiverate of 20%) on their independent test set of 2100 labeled images. Medical CBIR is now, however,dominated by deep learning algorithms [311, 312, 313]. As an example, by retrieving medical casessimilar to a given image, Pizarro et al. [279] developed a CNN for automatically inferring thecontrast of MRI scans based on the image intensity of multiple slices.

43See “search by image” https://images.google.com, https://developers.google.com/custom-search, and alsohttps://tineye.com, indexing more than 30 billion images

21

Recently, deep learning methods have also been used for automated generation of radiology reports,typically incorporating long-short-term-memory (LSTM) network models to generate the textualparagraphs [314, 315, 316, 317], and also to identify findings in radiology reports [318, 319, 320].

4. Open science and reproducible research in machine learning for medical imaging

Machine learning is moving at a breakneck speed, too fast for the standard peer-review processto keep up. Many of the most celebrated and impactful papers in machine learning over the pastfew years are only available as preprints, or published in conference proceedings long after theirresults are well-known and incorporated in the research of others. Bypassing peer-review has somedownsides, of course, but these are somewhat mitigated by researchers’ willingness to share codeand data.44

Most of the main new ideas and methods are posted to the arXiv preprint server45, and theaccompanying code shared on the GitHub platform46. The data sets used are often openly availablethrough various repositories. This, in addition to the many excellent online educational resources47,makes it easy to get started in the field. Select a problem you find interesting based on openlyavailable data, a method described in a preprint, and an implementation uploaded to GitHub. Thisforms a good starting point for an interesting machine learning project.

Another interesting aspect about modern machine learning and data science is the prevalence ofcompetitions, with the now annual ImageNet Large Scale Visual Recognition Challenge (ILSVRC)competition as the main driver of progress in deep learning for computer vision since 2012. Eachcompetition typically draws large number of participants, and the top results often push the state-of-the art to a new level. In addition to inspiring new ideas, competitions also provide natural entrypoints to modern machine learning. It is interesting to note how deep learning-based models arecompletely dominating the leaderboards of essentially all image-based competitions. Other machinelearning models, or non-machine learning-based techniques, have largely been outclassed.

What’s true about the openness of machine learning in general is increasingly true also for thesub-field of machine learning for medical image analysis. We’ve listed a few examples of openlyavailable implementations, data sets and challenges in tables 3, 4 and 5 below.

Table 3: A short list of openly available code for ML in medical imaging

Summary Reference Implementation

NiftyNet. An open source convolutionalneural networks platform for medical im-age analysis and image-guided therapy

[321, 322] http://niftynet.io

DLTK. State of the art reference imple-mentations for deep learning on medicalimages

[132] https://github.com/DLTK/DLTK

44In the spirit of sharing and open science, we’ve created a GitHub repository to accompany our article, availableat https://github.com/MMIV-ML/DLMI2018.

45http://arxiv.org46https://github.com47For example http://www.fast.ai, https://www.deeplearning.ai, http://cs231n.stanford.edu, https://

developers.google.com/machine-learning/crash-course

22

DeepMedic [323] https://github.com/Kamnitsask/

deepmedic

U-Net: Convolutional Networks forBiomedical Image Segmentation

[324] https://lmb.informatik.uni-freiburg.

de/people/ronneber/u-net

V-net [85] https://github.com/faustomilletari/

VNet

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust SemanticPixel-Wise Labelling

[325] https://mi.eng.cam.ac.uk/projects/

segnet

Brain lesion synthesis using GANs [213] https://github.com/khcs/

brain-synthesis-lesion-segmentation

GANCS: Compressed Sensing MRI basedon Deep Generative Adversarial Network

[326] https://github.com/gongenhao/GANCS

Deep MRI Reconstruction [136] https://github.com/js3611/

Deep-MRI-Reconstruction

Graph Convolutional Networks for brainanalysis in populations, combining imag-ing and non-imaging data

[327] https://github.com/parisots/

population-gcn

Table 4: A short list of medical imaging data sets and repositories

Name Summary Link

OpenNeuro An open platform for sharing neuroimag-ing data under the public domain license.Contains brain images from 168 studies(4,718 participants) with various imagingmodalities and acquisition protocols.

https://openneuro.org48

UK Biobank Health data from half a million partici-pants. Contains MRI images from 15.000participants, aiming to reach 100.000.

http://www.ukbiobank.ac.uk/

TCIA The cancer imaging archive hosts a largearchive of medical images of cancer ac-cessible for public download. Currentlycontains images from from 14.355 patientsacross 77 collections.

http://www.cancerimagingarchive.net

ABIDE The autism brain imaging data exchange.Contains 1114 datasets from 521 individ-uals with Autism Spectrum Disorder and593 controls.

http://fcon_1000.projects.nitrc.org/

indi/abide

48Data can be downloaded from the AWS S3 Bucket https://registry.opendata.aws/openneuro.

23

ADNI The Alzheimer’s disease neuroimaging ini-tiative. Contains image data from almost2000 participants (controls, early MCI,MCI, late MCI, AD)

http://adni.loni.usc.edu/

Table 5: A short list of medical imaging competitions

Name Summary Link

Grand-Challenges Grand challenges in biomedi-cal image analysis. Hosts andlists a large number of compe-titions

https://grand-challenge.org/

RSNA Pneumonia Detec-tion Challenge

Automatically locate lungopacities on chest radiographs

https://www.kaggle.com/c/

rsna-pneumonia-detection-challenge

HVSMR 2016 Segment the blood pool andmyocardium from a 3D cardio-vascular magnetic resonanceimage

http://segchd.csail.mit.edu/

ISLES 2018 Ischemic Stroke Lesion Seg-mentation 2018. The goal isto segment stroke lesions basedon acute CT perfusion data.

http://www.isles-challenge.org/

BraTS 2018 Multimodal Brain Tumor Seg-mentation. The goal is to seg-ment brain tumors in multi-modal MRI scans.

http://www.med.upenn.edu/sbia/

brats2018.html

CAMELYON17 The goal is to develop al-gorithms for automated de-tection and classification ofbreast cancer metastases inwhole-slide images of histolog-ical lymph node sections.

https://camelyon17.

grand-challenge.org/Home

ISIC 2018 Skin Lesion Analysis TowardsMelanoma Detection

https://challenge2018.

isic-archive.com/

Kaggle’s 2018 Data ScienceBowl

Spot Nuclei. Speed Cures. https://www.kaggle.com/c/

data-science-bowl-2018

Kaggle’s 2017 Data ScienceBowl

Turning Machine IntelligenceAgainst Lung Cancer

https://www.kaggle.com/c/

data-science-bowl-2017

Kaggle’s 2016 Data ScienceBowl

Transforming How We Diag-nose Heart Disease

https://www.kaggle.com/c/

second-annual-data-science-bowl

MURA Determine whether a bone X-ray is normal or abnormal

https://stanfordmlgroup.github.io/

competitions/mura/

24

5. Challenges, limitations and future perspectives

It is clear that deep neural networks are very useful when one is tasked with producing accuratedecisions based on complicated data sets. But they come with some significant challenges andlimitations that you either have to accept or try to overcome. Some are general: from technicalchallenges related to the lack of mathematical and theoretical underpinnings of many central deeplearning models and techniques, and the resulting difficulty in deciding exactly what it is that makesone model better than another, to societal challenges related to maximization and spread of thetechnological benefits [328, 329] and the problems related to the tremendous amounts of hype andexcitement49. Others are more domain-specific.

In deep learning for standard computer vision tasks, like object recognition and localization,powerful models and a set of best practices have been developed over the last few years. The paceof development is still incredibly high, but certain things seem to be settled, at least momentarily.Using the basic building blocks described above, placed according to the ideas behind, say, ResNetand SENet, will easily result in close to state-of-the-art performance on two-dimensional objectdetection, image classification and segmentation tasks.

However, the story for deep learning in medical imaging is not quite as settled. One issue is thatmedical images are often three-dimensional, and three-dimensional convolutional neural networksare as well-developed as their 2D counterparts. One quickly meet challenges associated to memoryand compute consumption when using CNNs with higher-dimensional image data, challenges thatresearchers are trying various approaches to deal with (treating 3D as stacks of 2Ds, patch- orsegment-based training and inference, downscaling, etc). It is clear that the ideas behind state-of-the-art two-dimensional CNNs can be lifted to three dimensions, but also that adding a thirdspatial dimension results in additional constraints. Other important challenges are related to data,trust, interpretability, workflow integration, and regulations, as discussed below.

5.1. Data

This is a crucially important obstacle for deep neural networks, especially in medical dataanalysis. When deploying deep neural networks, or any other machine learning model, one isinstantly faced with challenges related to data access, privacy issues, data protection, and more.

As privacy and data protection is often a requirement when dealing with medical data, newtechniques for training models without exposing the underlying training data to the user of themodel are necessary. It is not enough to merely restrict access to the training set used to constructthe model, as it is easy to use the model itself to discover details about the training set [330]. Evenhiding the model and only exposing a prediction interface would still leave it open to attack, forexample in the form of model-inversion [331] and membership attacks [332]. Most current work ondeep learning for medical data analysis use either open, anonymized data sets (as those in Table4), or locally obtained anonymized research data, making these issues less relevant. However, thegeneral deep learning community are focusing a lot of attention on the issue of privacy, and newtechniques and frameworks for federated learning [333]50, split learning [334, 335] and differentialprivacy [336, 337, 338] are rapidly improving. See [339] for a recent survey. There are a fewexamples of these ideas entering the medical machine learning community, as in [340] where thedistribution of deep learning models among several medical institutions was investigated, but then

49Lipton: Machine Learning: The Opportunity and the Opportunists https://www.technologyreview.com/video/

612109, Jordan: Artificial Intelligence – The Revolution Hasn’t Happened Yet https://medium.com/@mijordan3/

artificial-intelligence-the-revolution-hasnt-happened-yet-5e1d5812e1e750See for example https://ai.googleblog.com/2017/04/federated-learning-collaborative.html

25

without considering the above privacy issues. As machine learning systems in medicine grows tolarger scales, perhaps even including computations and learning on the “edge”, federated learningand differential privacy will likely become the focus of much research in our community.

If you are able to surmount these obstacles, you will be confronted with deep neural networks’insatiable appetite for training data. These are very inefficient models, requiring large numberof training samples before they can produce anything remotely useful, and labeled training datais typically both expensive and difficult to produce. In addition, the training data has to berepresentative of the data the network will meet in the future. If the training samples are froma data distribution that is very different from the one met in the real world, then the network’sgeneralization performance will be lower than expected. See [341] for a recent exploration of thisissue. Considering the large difference between the high-quality images one typically work withwhen doing research and the messiness of the real, clinical world, this can be a major obstacle whenputting deep learning systems into production.

Luckily there are ways to alleviate these problems somewhat. A widely used technique is transferlearning, also called fine-tuning or pre-training: first you train a network to perform a task wherethere is an abundance of data, and then you copy weights from this network to a network designedfor the task at hand. For two-dimensional images one will almost always use a network that hasbeen pre-trained on the ImageNet data set. The basic features in the earlier layers of the neuralnetwork found from this data set typically retain their usefulness in any other image-related task(or are at least form a better starting point than random initialization of the weights, which isthe alternative). Starting from weights tuned on a larger training data set can also make thenetwork more robust. Focusing the weight updates during training on later layers requires lessdata than having to do significant updates throughout the entire network. One can also do inter-organ transfer learning in 3D, an idea we have used for kidney segmentation, where pre-traininga network to do brain segmentation decreased the number of annotated kidneys needed to achievegood segmentation performance [342]. The idea of pre-training networks is not restricted to images.Pre-training entire models has recently been demonstrated to greatly impact the performance ofnatural language processing systems [3, 4, 5].

Another widely used technique is augmenting the training data set by applying various trans-formations that preserves the labels, as in rotations, scalings and intensity shifts of images, or moreadvanced data augmentation techniques like anatomically sound deformations, or other data setspecific operations (for example in our work on kidney segmentation from DCE-MRI, where weused image registration to propagate labels through a time course of images [343]). Data synthesis,as in [213], is another interesting approach.

In short, as expert annotators are expensive, or simply not available, spending large computa-tional resources to expand your labeled training data set, e.g. indirectly through transfer learningor directly through data augmentation, is typically worthwhile. But whatever you do, the way cur-rent deep neural networks are constructed and trained results in significant data size requirements.There are new ways of constructing more data-efficient deep neural networks on the horizon, for ex-ample by encoding more domain-specific elements in the neural network structure as in the capsulesystems of [344, 345], which adds viewpoint invariance. It is also possible to add attention mecha-nisms to neural networks [346, 347], enabling them to focus their resources on the most informativecomponents of each layer input.

However, the networks that are most frequently used, and with the best raw performance, remainthe data-hungry standard deep neural networks.

26

5.2. Interpretability, trust and safety

As deep neural networks relies on complicated interconnected hierarchical representations of thetraining data to produce its predictions, interpreting these predictions becomes very difficult. Thisis the “black box” problem of deep neural networks [348]. They are capable of producing extremelyaccurate predictions, but how can you trust predictions based on features you cannot understand?Considerable effort goes into developing new ways to deal with this problem, including DARPAlaunching a whole program “Explainable AI ”51 dedicated to this issue, and lots of research goinginto enhancing interpretability [349, 350], and finding new ways to measure sensitivity and visualizefeatures [68, 351, 352, 353? ].

Another way to increase their trustworthiness is to make them produce robust uncertaintyestimates in addition to predictions. The field of Bayesian Deep Learning aims to combine deeplearning and Bayesian approaches to uncertainty. The ideas date back to the early 90s [354, 355, 356],but the field has recently seen renewed interest from the machine learning community at large, asnew ways of computing uncertainty estimates from state of the art deep learning models have beendeveloped [59, 65, 357]. In addition to producing valuable measures that function as uncertaintymeasures [358, 359, 66], these techniques can also lessen deep neural networks susceptibility toadversarial attacks [357, 360].

5.3. Workflow integration, regulations

Another stumbling block for successful incorporation of deep learning methods is workflow in-tegration. It is possible to end up developing clever machine learning system for clinical use thatturn out to be practically useless for actual clinicians. Attempting to augment already establishedprocedures necessitates knowledge of the entire workflow. Involving the end-user in the process ofcreating and evaluating systems can make this a little less of an issue, and can also increase the endusers’ trust in the systems52, as you can establish a feedback loop during the development process.But still, even if there is interest on the “ground floor” and one is able to get prototype systemsinto the hands of clinicians, there are many higher-ups to convince and regulatory, ethical and legalhurdles to overcome.

5.4. Perspectives and future expectations

Deep learning in medical data analysis is here to stay. Even though there are many challengesassociated to the introduction of deep learning in clinical settings, the methods produce resultsthat are too valuable to discard. This is illustrated by the tremendous amounts of high-impactpublications in top-journals dealing with deep learning in medical imaging (for example [17, 21, 30,32, 40, 90, 162, 137, 140, 273, 285], all published in 2018). As machine learning researchers andpractitioners gain more experience, it will become easier to classify problems according to whatsolution approach is the most reasonable: (i) best approached using deep learning techniques end-to-end, (ii) best tackled by a combination of deep learning with other techniques, or (iii) no deeplearning component at all.

Beyond the application of machine learning in medical imaging, we believe that the attentionin the medical community can also be leveraged to strengthen the general computational mindsetamong medical researchers and practitioners, mainstreaming the field of computational medicine53.Once there are enough high-impact software-systems based on mathematics, computer science,

51https://www.darpa.mil/program/explainable-artificial-intelligence52The approach we have taken at our MMIV center https://mmiv.no, located inside the Department of Radiology53In-line with the ideas of the convergence of disciplines and the “future of health”, as described in [361]

27

physics and engineering entering the daily workflow in the clinic, the acceptance for other suchsystems will likely grow. The access to bio-sensors and (edge) computing on wearable devices formonitoring disease or lifestyle, plus an ecosystem of machine learning and other computationalmedicine-based technologies, will then likely facilitate the transition to a new medical paradigmthat is predictive, preventive, personalized, and participatory - P4 medicine [362]54.

Acknowledgements

We thank Renate Gruner for useful discussions. The anonymous reviewers gave us excellentconstructive feedback that led to several improvements throughout the article. Our work wasfinancially supported by the Bergen Research Foundation through the project “Computationalmedical imaging and machine learning – methods, infrastructure and applications”.

References

References

[1] A. Krizhevsky, I. Sutskever, G. E. Hinton, Ima-geNet classification with deep convolutional neuralnetworks, in: F. Pereira, C. J. C. Burges, L. Bottou,K. Q. Weinberger (Eds.), Advances in Neural Infor-mation Processing Systems 25, Curran Associates,Inc., 2012, pp. 1097–1105.

[2] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Per-ona, D. Ramanan, P. Dollar, C. L. Zitnick, Mi-crosoft COCO: Common objects in context, in: Eu-ropean Conference on Computer Vision, Springer,pp. 740–755.

[3] M. Peters, M. Neumann, M. Iyyer, M. Gardner,C. Clark, K. Lee, L. Zettlemoyer, Deep contextu-alized word representations, in: Proceedings of the2018 Conference of the North American Chapter ofthe Association for Computational Linguistics: Hu-man Language Technologies, Volume 1 (Long Pa-pers), volume 1, pp. 2227–2237.

[4] J. Howard, S. Ruder, Universal language model fine-tuning for text classification, in: Proceedings of the56th Annual Meeting of the Association for Compu-tational Linguistics (Volume 1: Long Papers), vol-ume 1, pp. 328–339.

[5] A. Radford, K. Narasimhan, T. Salimans,I. Sutskever, Improving language understand-ing by generative pre-training (2018).

[6] W. Xiong, L. Wu, F. Alleva, J. Droppo, X. Huang,A. Stolcke, The Microsoft 2017 Conversationalspeech recognition system, in: Proc. Speech andSignal Processing (ICASSP) 2018 IEEE Int. Conf.Acoustics, pp. 5934–5938.

[7] A. van den Oord, S. Dieleman, H. Zen, K. Si-monyan, O. Vinyals, A. Graves, N. Kalchbren-ner, A. Senior, K. Kavukcuoglu, WaveNet: Agenerative model for raw audio, arXiv preprintarXiv:1609.03499v2 (2016).

[8] C. Guo, F. Berkhahn, Entity embeddings of cate-gorical variables, arXiv preprint arXiv:1604.06737(2016).

[9] A. De Brebisson, E. Simon, A. Auvolat, P. Vin-cent, Y. Bengio, Artificial neural networks ap-plied to taxi destination prediction, arXiv preprintarXiv:1508.00021 (2015).

[10] D. George, E. Huerta, Deep learning for real-timegravitational wave detection and parameter estima-tion: Results with advanced LIGO data, PhysicsLetters B 778 (2018) 64–70.

[11] D. George, H. Shen, E. Huerta, Classification andunsupervised clustering of LIGO data with deeptransfer learning, Physical Review D 97 (2018)101501.

[12] H. Shen, D. George, E. Huerta, Z. Zhao, Denois-ing gravitational waves using deep learning withrecurrent denoising autoencoders, arXiv preprintarXiv:1711.09919 (2017).

[13] M. Raissi, G. E. Karniadakis, Hidden physics mod-els: Machine learning of nonlinear partial differen-tial equations, Journal of Computational Physics357 (2018) 125–141.

54http://p4mi.org

28

[14] A. Karpatne, G. Atluri, J. H. Faghmous, M. Stein-bach, A. Banerjee, A. Ganguly, S. Shekhar, N. Sam-atova, V. Kumar, Theory-guided data science: Anew paradigm for scientific discovery from data,IEEE Transactions on Knowledge and Data Engi-neering 29 (2017) 2318–2331.

[15] Gartner, Top Strategic Technology Trends for 2018,2018.

[16] D. Ravi, C. Wong, F. Deligianni, M. Berthelot,J. Andreu-Perez, B. Lo, G.-Z. Yang, Deep learningfor health informatics., IEEE journal of biomedicaland health informatics 21 (2017) 4–21.

[17] N. Ganapathy, R. Swaminathan, T. M. Deserno,Deep learning on 1-D biosignals: a taxonomy-basedsurvey, Yearbook of medical informatics 27 (2018)98–109.

[18] L. Kuhlmann, K. Lehnertz, M. P. Richardson,B. Schelter, H. P. Zaveri, Seizure prediction - readyfor a new era, Nature reviews. Neurology (2018).

[19] J.-M. Kwon, Y. Lee, Y. Lee, S. Lee, J. Park, Analgorithm based on deep learning for predicting in-hospital cardiac arrest, Journal of the AmericanHeart Association 7 (2018).

[20] H.-C. Shin, H. R. Roth, M. Gao, L. Lu, Z. Xu,I. Nogues, J. Yao, D. Mollura, R. M. Summers,Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset charac-teristics and transfer learning., IEEE transactionson medical imaging 35 (2016) 1285–1298.

[21] D. S. Kermany, M. Goldbaum, W. Cai, C. C. S.Valentim, H. Liang, S. L. Baxter, A. McKeown,G. Yang, X. Wu, F. Yan, J. Dong, M. K. Prasadha,J. Pei, M. Y. L. Ting, J. Zhu, C. Li, S. Hewett,J. Dong, I. Ziyar, A. Shi, R. Zhang, L. Zheng,R. Hou, W. Shi, X. Fu, Y. Duan, V. A. N. Huu,C. Wen, E. D. Zhang, C. L. Zhang, O. Li, X. Wang,M. A. Singer, X. Sun, J. Xu, A. Tafreshi, M. A.Lewis, H. Xia, K. Zhang, Identifying medical di-agnoses and treatable diseases by image-based deeplearning, Cell 172 (2018) 1122–1131.e9.

[22] J. L. Katzman, U. Shaham, A. Cloninger, J. Bates,T. Jiang, Y. Kluger, DeepSurv: personalized treat-ment recommender system using a Cox proportionalhazards deep neural network, BMC medical re-search methodology 18 (2018) 24.

[23] J. Jimenez, M. Skalic, G. Martınez-Rosell, G. DeFabritiis, KDEEP: Protein-Ligand absolute bind-ing affinity prediction via 3D-Convolutional Neu-ral Networks, Journal of Chemical Information andModeling 58 (2018) 287–296.

[24] A. A. Kalinin, G. A. Higgins, N. Reamaroon,S. Soroushmehr, A. Allyn-Feuer, I. D. Dinov, K. Na-jarian, B. D. Athey, Deep learning in pharmacoge-nomics: from gene regulation to patient stratifica-tion, Pharmacogenomics 19 (2018) 629–650.

[25] S. Jiang, K.-S. Chin, K. L. Tsui, A universal deeplearning approach for modeling the flow of patientsunder different severities, Computer methods andprograms in biomedicine 154 (2018) 191–203.

[26] K. C. Vranas, J. K. Jopling, T. E. Sweeney, M. C.Ramsey, A. S. Milstein, C. G. Slatore, G. J. Esco-bar, V. X. Liu, Identifying distinct subgroups of icupatients: A machine learning approach., Criticalcare medicine 45 (2017) 1607–1615.

[27] A. Rajkomar, E. Oren, K. Chen, A. M. Dai, N. Ha-jaj, M. Hardt, P. J. Liu, X. Liu, J. Marcus, M. Sun,et al., Scalable and accurate deep learning with elec-tronic health records, npj Digital Medicine 1 (2018)18.

[28] B. Shickel, P. J. Tighe, A. Bihorac, P. Rashidi, DeepEHR: A Survey of Recent Advances in Deep Learn-ing Techniques for Electronic Health Record (EHR)Analysis, IEEE Journal of Biomedical and HealthInformatics (2017).

[29] V. Gulshan, L. Peng, M. Coram, M. C. Stumpe,D. Wu, A. Narayanaswamy, S. Venugopalan,K. Widner, T. Madams, J. Cuadros, et al., Develop-ment and validation of a deep learning algorithm fordetection of diabetic retinopathy in retinal fundusphotographs, Jama 316 (2016) 2402–2410.

[30] R. Poplin, A. V. Varadarajan, K. Blumer, Y. Liu,M. McConnell, G. Corrado, L. Peng, D. Webster,Predicting Cardiovascular Risk Factors in RetinalFundus Photographs using Deep Learning, NatureBiomedical Engineering (2018).

[31] R. Poplin, P.-C. Chang, D. Alexander, S. Schwartz,T. Colthurst, A. Ku, D. Newburger, J. Dijamco,N. Nguyen, P. T. Afshar, S. S. Gross, L. Dorfman,C. Y. McLean, M. A. DePristo, A universal SNPand small-indel variant caller using deep neural net-works, Nature Biotechnology (2018).

[32] J. De Fauw, J. R. Ledsam, B. Romera-Paredes,S. Nikolov, N. Tomasev, S. Blackwell, H. Askham,X. Glorot, B. O’Donoghue, D. Visentin, et al., Clin-ically applicable deep learning for diagnosis and re-ferral in retinal disease, Nature medicine 24 (2018)1342.

[33] Y. Qin, K. Kamnitsas, S. Ancha, J. Nanavati,G. Cottrell, A. Criminisi, A. Nori, AutofocusLayer for Semantic Segmentation, arXiv preprintarXiv:1805.08403 (2018).

29

[34] K. Kamnitsas, C. Baumgartner, C. Ledig, V. New-combe, J. Simpson, A. Kane, D. Menon, A. Nori,A. Criminisi, D. Rueckert, et al., Unsupervised do-main adaptation in brain lesion segmentation withadversarial networks, in: International Confer-ence on Information Processing in Medical Imaging,Springer, pp. 597–609.

[35] C. Xiao, E. Choi, J. Sun, Opportunities and chal-lenges in developing deep learning models usingelectronic health records data: a systematic review,Journal of the American Medical Informatics Asso-ciation (2018).

[36] D. Silver, J. Schrittwieser, K. Simonyan,I. Antonoglou, A. Huang, A. Guez, T. Hu-bert, L. Baker, M. Lai, A. Bolton, et al., Masteringthe game of Go without human knowledge, Nature550 (2017) 354.

[37] A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M.Swetter, H. M. Blau, S. Thrun, Dermatologist-levelclassification of skin cancer with deep neural net-works, Nature 542 (2017) 115–118.

[38] R. Poplin, A. V. Varadarajan, K. Blumer, Y. Liu,M. V. McConnell, G. S. Corrado, L. Peng, D. R.Webster, Prediction of cardiovascular risk factorsfrom retinal fundus photographs via deep learning,Nature Biomedical Engineering 2 (2018) 158.

[39] Y. LeCun, Y. Bengio, G. Hinton, Deep learning,nature 521 (2015) 436.

[40] G. Hinton, Deep Learning A Technology With thePotential to Transform Health Care (2018) 1–2.

[41] I. Goodfellow, Y. Bengio, A. Courville, Deep Learn-ing, MIT Press, 2016. www.deeplearningbook.org.

[42] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Se-tio, F. Ciompi, M. Ghafoorian, J. A. W. M. van derLaak, B. van Ginneken, C. I. Snchez, A survey ondeep learning in medical image analysis., Medicalimage analysis 42 (2017) 60–88.

[43] A. H. Marblestone, G. Wayne, K. P. Kording, To-ward an Integration of Deep Learning and Neuro-science, Frontiers in Computational Neuroscience10 (2016) 94.

[44] D. Hassabis, D. Kumaran, C. Summerfield,M. Botvinick, Neuroscience-Inspired Artificial In-telligence, Neuron 95 (2017) 245–258.

[45] A. Banino, C. Barry, B. Uria, C. Blundell, T. Lil-licrap, P. Mirowski, A. Pritzel, M. J. Chadwick,T. Degris, J. Modayil, et al., Vector-based nav-igation using grid-like representations in artificialagents, Nature 557 (2018) 429.

[46] G. Cybenko, Approximation by superpositions of asigmoidal function, Mathematics of control, signalsand systems 2 (1989) 303–314.

[47] K. Hornik, M. Stinchcombe, H. White, Multilayerfeedforward networks are universal approximators,Neural networks 2 (1989) 359–366.

[48] M. Leshno, V. Y. Lin, A. Pinkus, S. Schocken, Mul-tilayer feedforward networks with a nonpolynomialactivation function can approximate any function,Neural networks 6 (1993) 861–867.

[49] S. Sonoda, N. Murata, Neural network with un-bounded activation functions is universal approxi-mator, Applied and Computational Harmonic Anal-ysis 43 (2017) 233–268.

[50] M. A. Nielsen, Neural networks and deep learning,Determination Press, 2015.

[51] C. C. Aggarwal, Neural networks and deep learning,Springer, 2018.

[52] F. Rosenblatt, The perceptron: a probabilisticmodel for information storage and organization inthe brain., Psychological review 65 (1958) 386.

[53] S. Linnainmaa, The representation of the cumu-lative rounding error of an algorithm as a taylorexpansion of the local rounding errors, Master’sThesis (in Finnish), Univ. Helsinki (1970) 6–7.

[54] P. Werbos, Beyond regression: New tools for pre-diction and analysis in the behavioral sciences, Ph.D. dissertation, Harvard University (1974).

[55] D. E. Rumelhart, G. E. Hinton, R. J. Williams,Learning representations by back-propagating er-rors, nature 323 (1986) 533.

[56] A. Cauchy, Methode generale pour la resolution dessystemes dequations simultanees, Comp. Rend. Sci.Paris 25 (1847) 536–538.

[57] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner,Gradient-based learning applied to document recog-nition, Proceedings of the IEEE 86 (1998) 2278–2324.

[58] S. C. Lo, M. T. Freedman, J. S. Lin, S. K.Mun, Automatic lung nodule detection using pro-file matching and back-propagation neural networktechniques., Journal of digital imaging 6 (1993) 48–54.

[59] S. Murray, An exploratory analysis of multi-classuncertainty approximation in Bayesian convolu-tion neural networks, Master’s thesis, University ofBergen, 2018.

30

[60] D.-A. Clevert, T. Unterthiner, S. Hochreiter, Fastand accurate deep network learning by exponentiallinear units (elus), arXiv preprint arXiv:1511.07289(2015).

[61] K. He, X. Zhang, S. Ren, J. Sun, Delving deepinto rectifiers: Surpassing human-level performanceon imagenet classification, in: Proceedings of theIEEE international conference on computer vision,pp. 1026–1034.

[62] J. T. Springenberg, A. Dosovitskiy, T. Brox,M. Riedmiller, Striving for simplicity: The allconvolutional net, arXiv preprint arXiv:1412.6806(2014).

[63] N. Srivastava, G. Hinton, A. Krizhevsky,I. Sutskever, R. Salakhutdinov, Dropout: asimple way to prevent neural networks from over-fitting, The Journal of Machine Learning Research15 (2014) 1929–1958.

[64] K. Rashmi, R. Gilad-Bachrach, Dart: Dropoutsmeet multiple additive regression trees, in: Inter-national Conference on Artificial Intelligence andStatistics, pp. 489–497.

[65] Y. Gal, Uncertainty in deep learning, Ph.D. thesis,University of Cambridge, 2016.

[66] K. Wickstrøm, M. Kampffmeyer, R. Jenssen, Un-certainty Modeling and Interpretability in Convolu-tional Neural Networks for Polyp Segmentation, in:2018 IEEE 28th International Workshop on MachineLearning for Signal Processing (MLSP), IEEE, pp.1–6.

[67] S. Ioffe, C. Szegedy, Batch Normalization: Acceler-ating Deep Network Training by Reducing InternalCovariate Shift, in: International Conference onMachine Learning, pp. 448–456.

[68] M. D. Zeiler, R. Fergus, Visualizing and under-standing convolutional networks, in: European con-ference on computer vision, Springer, pp. 818–833.

[69] K. Simonyan, A. Zisserman, Very deep convolu-tional networks for large-scale image recognition,arXiv preprint arXiv:1409.1556 (2014).

[70] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabi-novich, Going deeper with convolutions, in: Pro-ceedings of the IEEE conference on computer visionand pattern recognition, pp. 1–9.

[71] M. Lin, Q. Chen, S. Yan, Network in network, arXivpreprint arXiv:1312.4400 (2013).

[72] K. He, X. Zhang, S. Ren, J. Sun, Deep residuallearning for image recognition, in: Proceedings ofthe IEEE conference on computer vision and pat-tern recognition, pp. 770–778.

[73] R. K. Srivastava, K. Greff, J. Schmidhuber, Train-ing very deep networks, in: Advances in neuralinformation processing systems, pp. 2377–2385.

[74] G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Wein-berger, Densely connected convolutional networks,in: CVPR, volume 1, p. 3.

[75] S. Xie, R. Girshick, P. Dollar, Z. Tu, K. He, Ag-gregated residual transformations for deep neuralnetworks, in: Computer Vision and Pattern Recog-nition (CVPR), 2017 IEEE Conference on, IEEE,pp. 5987–5995.

[76] J. Hu, L. Shen, G. Sun, Squeeze-and-excitation net-works, arXiv preprint arXiv:1709.01507 7 (2017).

[77] B. Zoph, V. Vasudevan, J. Shlens, Q. V. Le, Learn-ing transferable architectures for scalable imagerecognition, arXiv preprint arXiv:1707.07012 2(2017).

[78] I. Bello, B. Zoph, V. Vasudevan, Q. V. Le, Neuraloptimizer search with reinforcement learning, in:D. Precup, Y. W. Teh (Eds.), Proceedings of the34th International Conference on Machine Learn-ing, volume 70 of Proceedings of Machine LearningResearch, PMLR, International Convention Centre,Sydney, Australia, 2017, pp. 459–468.

[79] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, Youonly look once: Unified, real-time object detection,in: Proceedings of the IEEE conference on com-puter vision and pattern recognition, pp. 779–788.

[80] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio,Generative Adversarial Nets, in: Z. Ghahramani,M. Welling, C. Cortes, N. D. Lawrence, K. Q. Wein-berger (Eds.), Advances in Neural Information Pro-cessing Systems 27, Curran Associates, Inc., 2014,pp. 2672–2680.

[81] G. Koch, R. Zemel, R. Salakhutdinov, Siamese neu-ral networks for one-shot image recognition, in:ICML Deep Learning Workshop, volume 2.

[82] J. Bromley, I. Guyon, Y. LeCun, E. Sackinger,R. Shah, Signature verification using a “siamese”time delay neural network, in: Advances in neuralinformation processing systems, pp. 737–744.

[83] O. Ronneberger, P. Fischer, T. Brox, U-net: Con-volutional networks for biomedical image segmenta-tion, in: International Conference on Medical im-age computing and computer-assisted intervention,Springer, pp. 234–241.

[84] J. Long, E. Shelhamer, T. Darrell, Fully convolu-tional networks for semantic segmentation, in: Pro-ceedings of the IEEE conference on computer visionand pattern recognition, pp. 3431–3440.

31

[85] F. Milletari, N. Navab, S.-A. Ahmadi, V-net: Fullyconvolutional neural networks for volumetric medi-cal image segmentation, in: 3D Vision (3DV), 2016Fourth International Conference on, IEEE, pp. 565–571.

[86] J. Redmon, Darknet: Open source neural networksin C, http://pjreddie.com/darknet/, 2013–2016.

[87] Y. H. Lee, Efficiency improvement in a busy ra-diology practice: Determination of musculoskeletalmagnetic resonance imaging protocol using deep-learning convolutional neural networks., Journal ofdigital imaging (2018).

[88] E. Gong, J. M. Pauly, M. Wintermark, G. Za-harchuk, Deep learning enables reduced gadoliniumdose for contrast-enhanced brain MRI, Journal ofmagnetic resonance imaging 48 (2018) 330–340.

[89] P. Meyer, V. Noblet, C. Mazzara, A. Lallement,Survey on deep learning for radiotherapy., Com-puters in biology and medicine 98 (2018) 126–146.

[90] F. Liu, H. Jang, R. Kijowski, T. Bradshaw, A. B.McMillan, Deep learning MR imaging-based atten-uation correction for PET/MR imaging, Radiology286 (2018) 676–684.

[91] A. Mehranian, H. Arabi, H. Zaidi, Vision 20/20:Magnetic resonance imaging-guided attenuationcorrection in pet/mri: Challenges, solutions, andopportunities., Medical physics 43 (2016) 1130–1155.

[92] J. Lao, Y. Chen, Z.-C. Li, Q. Li, J. Zhang, J. Liu,G. Zhai, A deep learning-based radiomics model forprediction of survival in glioblastoma multiforme.,Scientific reports 7 (2017) 10353.

[93] L. Oakden-Rayner, G. Carneiro, T. Bessen, J. C.Nascimento, A. P. Bradley, L. J. Palmer, Precisionradiology: Predicting longevity using feature engi-neering and deep learning methods in a radiomicsframework., Scientific reports 7 (2017) 1648.

[94] J. C. Peeken, M. Bernhofer, B. Wiestler, T. Gold-berg, D. Cremers, B. Rost, J. J. Wilkens, S. E.Combs, F. Nsslin, Radiomics in radiooncology -challenging the medical physicist., Physica medica48 (2018) 27–36.

[95] M. Izadyyazdanabadi, E. Belykh, M. A. Mooney,J. M. Eschbacher, P. Nakaji, Y. Yang, M. C. Preul,Prospects for theranostics in neurosurgical imag-ing: Empowering confocal laser endomicroscopy di-agnostics via deep learning., Frontiers in oncology8 (2018) 240.

[96] G. Haskins, J. Kruecker, U. Kruger, S. Xu, P. A.Pinto, B. J. Wood, P. Yan, Learning deep simi-larity metric for 3D MR-TRUS registration, arXivpreprint arXiv:1806.04548v1 (2018).

[97] X. Cao, J. Yang, J. Zhang, Q. Wang, P.-T. Yap,D. Shen, Deformable image registration using a cue-aware deep regression network, IEEE transactionson bio-medical engineering 65 (2018) 1900–1911.

[98] X. Yang, R. Kwitt, M. Styner, M. Niethammer,Quicksilver: Fast predictive image registration - adeep learning approach., NeuroImage 158 (2017)378–396.

[99] V. P. Kearney, S. Haaf, A. Sudhyadhom, G. Valdes,T. D. Solberg, An unsupervised convolutionalneural network-based algorithm for deformable im-age registration, Physics in medicine and biology(2018).

[100] J. Zheng, S. Miao, Z. Jane Wang, R. Liao, Pairwisedomain adaptation module for CNN-based 2-D/3-Dregistration., Journal of medical imaging (Belling-ham, Wash.) 5 (2018) 021204.

[101] E. M. A. Anas, P. Mousavi, P. Abolmaesumi, Adeep learning approach for real time prostate seg-mentation in freehand ultrasound guided biopsy,Medical image analysis 48 (2018) 107–116.

[102] T. Ching, D. S. Himmelstein, B. K. Beaulieu-Jones,A. A. Kalinin, B. T. Do, G. P. Way, E. Ferrero,P.-M. Agapow, M. Zietz, M. M. Hoffman, W. Xie,G. L. Rosen, B. J. Lengerich, J. Israeli, J. Lan-chantin, S. Woloszynek, A. E. Carpenter, A. Shriku-mar, J. Xu, E. M. Cofer, C. A. Lavender, S. C.Turaga, A. M. Alexandari, Z. Lu, D. J. Harris,D. DeCaprio, Y. Qi, A. Kundaje, Y. Peng, L. K.Wiley, M. H. S. Segler, S. M. Boca, S. J. Swami-dass, A. Huang, A. Gitter, C. S. Greene, Opportu-nities and obstacles for deep learning in biology andmedicine, Journal of the Royal Society, Interface 15(2018).

[103] J.-G. Lee, S. Jun, Y.-W. Cho, H. Lee, G. B. Kim,J. B. Seo, N. Kim, Deep learning in medical imag-ing: General overview., Korean journal of radiology18 (2017) 570–584.

[104] D. Rueckert, B. Glocker, B. Kainz, Learning clini-cally useful information from images: Past, presentand future., Medical image analysis 33 (2016) 13–18.

[105] G. Chartrand, P. M. Cheng, E. Vorontsov,M. Drozdzal, S. Turcotte, C. J. Pal, S. Kadoury,A. Tang, Deep learning: A primer for radiologists,Radiographics : a review publication of the Radi-ological Society of North America, Inc 37 (2017)2113–2131.

[106] B. J. Erickson, P. Korfiatis, Z. Akkus, T. L. Kline,Machine learning for medical imaging, Radiograph-ics : a review publication of the Radiological Societyof North America, Inc 37 (2017) 505–515.

32

[107] M. A. Mazurowski, M. Buda, A. Saha, M. R. Bashir,Deep learning in radiology: an overview of the con-cepts and a survey of the state of the art, arXivpreprint arXiv:1802.08717v1 (2018).

[108] M. P. McBee, O. A. Awan, A. T. Colucci, C. W.Ghobadi, N. Kadom, A. P. Kansagra, S. Tridanda-pani, W. F. Auffermann, Deep learning in radiol-ogy., Academic radiology (2018).

[109] P. Savadjiev, J. Chong, A. Dohan,M. Vakalopoulou, C. Reinhold, N. Paragios,B. Gallix, Demystification of AI-driven medicalimage interpretation: past, present and future.,European radiology (2018).

[110] J. H. Thrall, X. Li, Q. Li, C. Cruz, S. Do, K. Dreyer,J. Brink, Artificial intelligence and machine learn-ing in radiology: Opportunities, challenges, pitfalls,and criteria for success., Journal of the AmericanCollege of Radiology : JACR 15 (2018) 504–508.

[111] R. Yamashita, M. Nishio, R. K. G. Do, K. Togashi,Convolutional neural networks: an overview and ap-plication in radiology., Insights into imaging (2018).

[112] K. Yasaka, H. Akai, A. Kunimatsu, S. Kiryu,O. Abe, Deep learning with convolutional neuralnetwork in radiology., Japanese journal of radiol-ogy 36 (2018) 257–272.

[113] M. L. Giger, Machine learning in medical imag-ing, Journal of the American College of Radiology: JACR 15 (2018) 512–520.

[114] B. J. Erickson, P. Korfiatis, Z. Akkus, T. Kline,K. Philbrick, Toolkits and libraries for deep learn-ing, Journal of digital imaging 30 (2017) 400–405.

[115] G. Zaharchuk, E. Gong, M. Wintermark, D. Rubin,C. P. Langlotz, Deep learning in neuroradiology.,AJNR. American journal of neuroradiology (2018).

[116] Z. Akkus, A. Galimzianova, A. Hoogi, D. L. Rubin,B. J. Erickson, Deep learning for brain MRI seg-mentation: State of the art and future directions.,Journal of digital imaging 30 (2017) 449–459.

[117] E.-J. Lee, Y.-H. Kim, N. Kim, D.-W. Kang, Deepinto the brain: Artificial intelligence in stroke imag-ing., Journal of stroke 19 (2017) 277–285.

[118] R. Feng, M. Badgeley, J. Mocco, E. K. Oermann,Deep learning guided stroke management: a reviewof clinical applications, Journal of neurointerven-tional surgery 10 (2018) 358–362.

[119] S. Vieira, W. H. L. Pinaya, A. Mechelli, Using deeplearning to investigate the neuroimaging correlatesof psychiatric and neurological disorders: Methodsand applications., Neuroscience and biobehavioralreviews 74 (2017) 58–75.

[120] J. R. Burt, N. Torosdagli, N. Khosravan,H. RaviPrakash, A. Mortazi, F. Tissavirasingham,S. Hussein, U. Bagci, Deep learning beyond cats anddogs: recent advances in diagnosing breast cancerwith deep neural networks, The British journal ofradiology 91 (2018) 20170545.

[121] R. K. Samala, H.-P. Chan, L. M. Hadjiiski, M. A.Helvie, K. H. Cha, C. D. Richter, Multi-task trans-fer learning deep convolutional neural network: ap-plication to computer-aided diagnosis of breast can-cer on mammograms., Physics in medicine and bi-ology 62 (2017) 8894–8908.

[122] B. van Ginneken, Fifty years of computer analy-sis in chest imaging: rule-based, machine learning,deep learning, Radiological physics and technology10 (2017) 23–32.

[123] O. Morin, M. Vallires, A. Jochems, H. C. Woodruff,G. Valdes, S. E. Braunstein, J. E. Wildberger, J. E.Villanueva-Meyer, V. Kearney, S. S. Yom, T. D.Solberg, P. Lambin, A deep look into the futureof quantitative imaging in oncology: A statementof working principles and proposal for change., In-ternational journal of radiation oncology, biology,physics (2018).

[124] C. Parmar, J. D. Barry, A. Hosny, J. Quackenbush,H. J. W. L. Aerts, Data analysis strategies in med-ical imaging., Clinical cancer research : an officialjournal of the American Association for Cancer Re-search 24 (2018) 3492–3499.

[125] Y. Xue, S. Chen, J. Qin, Y. Liu, B. Huang, H. Chen,Application of deep learning in automated analysisof molecular images in cancer: A survey., Contrastmedia & molecular imaging 2017 (2017) 9512370.

[126] L. J. Brattain, B. A. Telfer, M. Dhyani, J. R. Grajo,A. E. Samir, Machine learning for medical ultra-sound: status, methods, and future opportunities,Abdominal radiology 43 (2018) 786–799.

[127] Q. Huang, F. Zhang, X. Li, Machine learning inultrasound computer-aided diagnostic systems: Asurvey, BioMed research international 2018 (2018)5137904.

[128] D. Shen, G. Wu, H.-I. Suk, Deep learning in med-ical image analysis., Annual review of biomedicalengineering 19 (2017) 221–248.

[129] K. Suzuki, Overview of deep learning in medicalimaging., Radiological physics and technology 10(2017) 257–273.

[130] C. Cao, F. Liu, H. Tan, D. Song, W. Shu, W. Li,Y. Zhou, X. Bo, Z. Xie, Deep learning and its ap-plications in biomedicine, Genomics, proteomics &bioinformatics 16 (2018) 17–32.

33

[131] P. Lakhani, D. L. Gray, C. R. Pett, P. Nagy,G. Shih, Hello world deep learning in medical imag-ing., Journal of digital imaging (2018).

[132] N. Pawlowski, S. I. Ktena, M. C. Lee, B. Kainz,D. Rueckert, B. Glocker, M. Rajchl, DLTK: State ofthe art reference implementations for deep learningon medical images, arXiv preprint arXiv:1711.06853(2017).

[133] Y. Yang, J. Sun, H. Li, Z. Xu, Deep ADMM-Net for compressive sensing MRI, in: D. D. Lee,M. Sugiyama, U. V. Luxburg, I. Guyon, R. Garnett(Eds.), Advances in Neural Information ProcessingSystems 29, Curran Associates, Inc., 2016, pp. 10–18.

[134] S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang,D. Feng, D. Liang, Accelerating magnetic resonanceimaging via deep learning, in: Biomedical Imaging(ISBI), 2016 IEEE 13th International Symposiumon, IEEE, pp. 514–517.

[135] C. Qin, J. V. Hajnal, D. Rueckert, J. Schlemper,J. Caballero, A. N. Price, Convolutional recurrentneural networks for dynamic mr image reconstruc-tion., IEEE transactions on medical imaging (2018).

[136] J. Schlemper, J. Caballero, J. V. Hajnal, A. N.Price, D. Rueckert, A deep cascade of convolutionalneural networks for dynamic MR image reconstruc-tion., IEEE transactions on medical imaging 37(2018) 491–503.

[137] F. Chen, V. Taviani, I. Malkiel, J. Y. Cheng, J. I.Tamir, J. Shaikh, S. T. Chang, C. J. Hardy, J. M.Pauly, S. S. Vasanawala, Variable-density single-shot fast Spin-Echo MRI with deep learning recon-struction by using variational networks, Radiology(2018) 180445.

[138] F. Knoll, K. Hammernik, E. Kobler, T. Pock, M. P.Recht, D. K. Sodickson, Assessment of the gener-alization of learned image reconstruction and thepotential for transfer learning, Magnetic resonancein medicine (2018).

[139] M. Mardani, E. Gong, J. Y. Cheng, S. S.Vasanawala, G. Zaharchuk, L. Xing, J. M. Pauly,Deep generative adversarial neural networks forcompressive sensing (GANCS) MRI., IEEE trans-actions on medical imaging (2018).

[140] B. Zhu, J. Z. Liu, S. F. Cauley, B. R. Rosen, M. S.Rosen, Image reconstruction by domain-transformmanifold learning., Nature 555 (2018) 487–492.

[141] T. Eo, Y. Jun, T. Kim, J. Jang, H.-J. Lee,D. Hwang, KIKI-net: cross-domain convolutionalneural networks for reconstructing undersampledmagnetic resonance images, Magnetic resonance inmedicine 80 (2018) 2188–2201.

[142] Y. Han, J. Yoo, H. H. Kim, H. J. Shin, K. Sung,J. C. Ye, Deep learning with domain adaptationfor accelerated projection-reconstruction MR, Mag-netic resonance in medicine 80 (2018) 1189–1205.

[143] J. Shi, Q. Liu, C. Wang, Q. Zhang, S. Ying, H. Xu,Super-resolution reconstruction of mr image with anovel residual learning network algorithm., Physicsin medicine and biology 63 (2018) 085011.

[144] G. Yang, S. Yu, H. Dong, G. Slabaugh, P. L.Dragotti, X. Ye, F. Liu, S. Arridge, J. Keegan,Y. Guo, D. Firmin, J. Keegan, G. Slabaugh, S. Ar-ridge, X. Ye, Y. Guo, S. Yu, F. Liu, D. Firmin,P. L. Dragotti, G. Yang, H. Dong, DAGAN: deepde-aliasing generative adversarial networks for fastcompressed sensing MRI reconstruction., IEEEtransactions on medical imaging 37 (2018) 1310–1321.

[145] A. Deistung, A. Schafer, F. Schweser, U. Bieder-mann, R. Turner, J. R. Reichenbach, Toward in vivohistology: a comparison of quantitative susceptibil-ity mapping (QSM) with magnitude-, phase-, andR2*-imaging at ultra-high magnetic field strength,Neuroimage 65 (2013) 299–314.

[146] A. Deistung, F. Schweser, J. R. Reichenbach,Overview of quantitative susceptibility mapping,NMR in Biomedicine 30 (2017).

[147] J. Yoon, E. Gong, I. Chatnuntawech, B. Bilgic,J. Lee, W. Jung, J. Ko, H. Jung, K. Setsompop,G. Zaharchuk, E. Y. Kim, J. Pauly, J. Lee, Quanti-tative susceptibility mapping using deep neural net-work: QSMnet., NeuroImage 179 (2018) 199–206.

[148] T. Liu, P. Spincemaille, L. De Rochefort,B. Kressler, Y. Wang, Calculation of susceptibil-ity through multiple orientation sampling (COS-MOS): a method for conditioning the inverse prob-lem from measured magnetic field map to suscepti-bility source image in MRI, Magnetic Resonance inMedicine 61 (2009) 196–204.

[149] K. G. B. Rasmussen, M. J. Kristensen, R. G.Blendal, L. R. Ostergaard, M. Plocharski,K. O’Brien, C. Langkammer, A. Janke, M. Barth,S. Bollmann, DeepQSM-Using Deep Learning toSolve the Dipole Inversion for MRI SusceptibilityMapping, Biorxiv (2018) 278036.

[150] D. Ma, V. Gulani, N. Seiberlich, K. Liu, J. L. Sun-shine, J. L. Duerk, M. A. Griswold, Magnetic reso-nance fingerprinting, Nature 495 (2013) 187–192.

[151] E. S. of Radiology (ESR), Magnetic resonance fin-gerprinting - a promising new approach to obtainstandardized imaging biomarkers from mri., In-sights into imaging 6 (2015) 163–165.

34

[152] D. L. Donoho, Compressed sensing, IEEE Transac-tions on Information Theory 52 (2006) 1289–1306.

[153] M. Lustig, D. Donoho, J. M. Pauly, Sparse mri:The application of compressed sensing for rapid mrimaging., Magnetic resonance in medicine 58 (2007)1182–1195.

[154] M. T. McCann, K. H. Jin, M. Unser, Convolutionalneural networks for inverse problems in imaging: Areview, IEEE Signal Processing Magazine 34 (2017)85–95.

[155] V. Shah, C. Hegde, Solving Linear Inverse Prob-lems Using GAN Priors: An Algorithm with Prov-able Guarantees, arXiv preprint arXiv:1802.08406(2018).

[156] A. Lucas, M. Iliadis, R. Molina, A. K. Katsaggelos,Using deep neural networks for inverse problems inimaging: beyond analytical methods, IEEE SignalProcessing Magazine 35 (2018) 20–36.

[157] H. K. Aggarwal, M. P. Mani, M. Jacob, MoDL:Model Based Deep Learning Architecture for In-verse Problems, IEEE transactions on medicalimaging (2018).

[158] H. Li, J. Schwab, S. Antholzer, M. Haltmeier,NETT: Solving Inverse Problems with Deep NeuralNetworks, arXiv preprint arXiv:1803.00092 (2018).

[159] D. Ma, Y. Jiang, Y. Chen, D. McGivney, B. Mehta,V. Gulani, M. Griswold, Fast 3D magnetic res-onance fingerprinting for a whole-brain coverage,Magnetic resonance in medicine 79 (2018) 2190–2197.

[160] T. Christen, N. A. Pannetier, W. W. Ni, D. Qiu,M. E. Moseley, N. Schuff, G. Zaharchuk, MR vas-cular fingerprinting: A new approach to computecerebral blood volume, mean vessel radius, and oxy-genation maps in the human brain, Neuroimage 89(2014) 262–270.

[161] B. Lemasson, N. Pannetier, N. Coquery, L. S. B.Boisserand, N. Collomb, N. Schuff, M. Moseley,G. Zaharchuk, E. L. Barbier, T. Christen, Mr vascu-lar fingerprinting in stroke and brain tumors mod-els., Scientific reports 6 (2016) 37071.

[162] B. Rieger, M. Akakaya, J. C. Pariente, S. Llufriu,E. Martinez-Heras, S. Weingrtner, L. R. Schad,Time efficient whole-brain coverage with MRfingerprinting using slice-interleaved echo-planar-imaging, Scientific reports 8 (2018) 6667.

[163] K. L. Wright, Y. Jiang, D. Ma, D. C. Noll, M. A.Griswold, V. Gulani, L. Hernandez-Garcia, Estima-tion of perfusion properties with mr fingerprintingarterial spin labeling., Magnetic resonance imaging50 (2018) 68–77.

[164] A. Panda, B. B. Mehta, S. Coppo, Y. Jiang, D. Ma,N. Seiberlich, M. A. Griswold, V. Gulani, Mag-netic resonance fingerprinting-an overview., Cur-rent opinion in biomedical engineering 3 (2017) 56–66.

[165] O. Cohen, B. Zhu, M. S. Rosen, MR fingerprintingdeep reconstruction network (DRONE), Magneticresonance in medicine 80 (2018) 885–894.

[166] E. Hoppe, G. Krzdrfer, T. Wrfl, J. Wetzl, F. Lu-gauer, J. Pfeuffer, A. Maier, Deep learning for mag-netic resonance fingerprinting: A new approach forpredicting quantitative parameter values from timeseries, Studies in health technology and informatics243 (2017) 202–206.

[167] J. Z. Bojorquez, S. Bricq, C. Acquitter, F. Brunotte,P. M. Walker, A. Lalande, What are normal relax-ation times of tissues at 3 T?, Magnetic resonanceimaging 35 (2017) 69–80.

[168] Z. Fang, Y. Chen, W. Lin, D. Shen, Quantificationof relaxation times in MR fingerprinting using deeplearning., Proceedings of the International Societyfor Magnetic Resonance in Medicine ... ScientificMeeting and Exhibition. International Society forMagnetic Resonance in Medicine. Scientific Meet-ing and Exhibition 25 (2017).

[169] P. Virtue, S. X. Yu, M. Lustig, Better than real:Complex-valued neural nets for MRI fingerprinting,in: Proc. IEEE Int. Conf. Image Processing (ICIP),pp. 3953–3957.

[170] M. Tygert, J. Bruna, S. Chintala, Y. LeCun, S. Pi-antino, A. Szlam, A mathematical motivationfor complex-valued convolutional networks, Neuralcomputation 28 (2016) 815–825.

[171] C. Trabelsi, O. Bilaniuk, Y. Zhang, D. Serdyuk,S. Subramanian, J. F. Santos, S. Mehri, N. Ros-tamzadeh, Y. Bengio, C. J. Pal, Deep complex net-works, arXiv preprint arXiv:1705.09792 (2017).

[172] J. Sijbers, A. J. den Dekker, J. Van Audekerke,M. Verhoye, D. Van Dyck, Estimation of the noise inmagnitude MR images., Magnetic resonance imag-ing 16 (1998) 87–90.

[173] E. R. McVeigh, R. M. Henkelman, M. J. Bronskill,Noise and filtration in magnetic resonance imaging.,Medical physics 12 (1985) 586–591.

[174] F. Baselice, G. Ferraioli, V. Pascazio, A. Sorriso,Bayesian mri denoising in complex domain, Mag-netic resonance imaging 38 (2017) 112–122.

[175] A. Phophalia, S. K. Mitra, 3d mr image denoisingusing rough set and kernel pca method., Magneticresonance imaging 36 (2017) 135–145.

35

[176] X. Zhang, Z. Xu, N. Jia, W. Yang, Q. Feng,W. Chen, Y. Feng, Denoising of 3D magnetic res-onance images by using higher-order singular valuedecomposition., Medical image analysis 19 (2015)75–86.

[177] D. Van De Ville, M. L. Seghier, F. Lazeyras, T. Blu,M. Unser, Wspm: wavelet-based statistical para-metric mapping., NeuroImage 37 (2007) 1205–1217.

[178] G. Salimi-Khorshidi, G. Douaud, C. F. Beckmann,M. F. Glasser, L. Griffanti, S. M. Smith, Automaticdenoising of functional MRI data: combining inde-pendent component analysis and hierarchical fusionof classifiers, Neuroimage 90 (2014) 449–468.

[179] M. Lysaker, A. Lundervold, X.-C. Tai, Noise re-moval using fourth-order partial differential equa-tion with applications to medical magnetic reso-nance images in space and time, IEEE transactionson image processing 12 (2003) 1579–1590.

[180] C. Bermudez, A. J. Plassard, T. L. Davis, A. T.Newton, S. M. Resnick, B. A. Landman, Learningimplicit brain MRI manifolds with deep learning,Proceedings of SPIE–the International Society forOptical Engineering 10574 (2018).

[181] A. Benou, R. Veksler, A. Friedman, T. Riklin Raviv,Ensemble of expert deep neural networks for spatio-temporal denoising of contrast-enhanced MRI se-quences, Medical image analysis 42 (2017) 145–159.

[182] Y. Gal, A. J. H. Mehnert, A. P. Bradley, K. McMa-hon, D. Kennedy, S. Crozier, Denoising of dynamiccontrast-enhanced MR images using dynamic non-local means, IEEE transactions on medical imaging29 (2010) 302–310.

[183] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio,P.-A. Manzagol, Stacked denoising autoencoders:learning useful representations in a deep networkwith a local denoising criterion, Journal of MachineLearning Research (JMLR) 11 (2010) 3371–3408.

[184] N. Dikaios, S. Arridge, V. Hamy, S. Punwani,D. Atkinson, Direct parametric reconstruction fromundersampled (k,t)-space data in dynamic contrastenhanced MRI, Medical image analysis 18 (2014)989–1001.

[185] Y. Guo, S. G. Lingala, Y. Zhu, R. M. Lebel, K. S.Nayak, Direct estimation of tracer-kinetic parame-ter maps from highly undersampled brain dynamiccontrast enhanced MRI, Magnetic resonance inmedicine 78 (2017) 1566–1578.

[186] S. P. Sourbron, D. L. Buckley, Classic mod-els for dynamic contrast-enhanced mri., NMR inbiomedicine 26 (2013) 1004–1027.

[187] V. Golkov, A. Dosovitskiy, J. I. Sperl, M. I. Men-zel, M. Czisch, P. Samann, T. Brox, D. Cremers, q-space deep learning: Twelve-fold shorter and model-free diffusion MRI scans, IEEE transactions onmedical imaging 35 (2016) 1344–1351.

[188] S. S. Gurbani, E. Schreibmann, A. A. Maudsley,J. S. Cordova, B. J. Soher, H. Poptani, G. Verma,P. B. Barker, H. Shim, L. A. D. Cooper, A con-volutional neural network to filter artifacts in spec-troscopic MRI, Magnetic resonance in medicine 80(2018) 1765–1775.

[189] S. P. Kyathanahally, A. Dring, R. Kreis, Deep learn-ing approaches for detection and removal of ghost-ing artifacts in MR spectroscopy, Magnetic reso-nance in medicine 80 (2018) 851–863.

[190] T. Kustner, A. Liebgott, L. Mauch, P. Martirosian,F. Bamberg, K. Nikolaou, B. Yang, F. Schick, S. Ga-tidis, Automated reference-free detection of motionartifacts in magnetic resonance images, MAGMA31 (2018) 243–256.

[191] L. Yue, H. Shen, J. Li, Q. Yuan, H. Zhang,L. Zhang, Image super-resolution: The techniques,applications, and future, Signal Processing 128(2016) 389–408.

[192] R. Z. Shilling, T. Q. Robbie, T. Bailloeul, K. Mewes,R. M. Mersereau, M. E. Brummer, A super-resolution framework for 3-d high-resolution andhigh-contrast imaging using 2-d multislice mri.,IEEE transactions on medical imaging 28 (2009)633–644.

[193] S. Ropele, F. Ebner, F. Fazekas, G. Reishofer,Super-resolution mri using microscopic spatial mod-ulation of magnetization., Magnetic resonance inmedicine 64 (2010) 1671–1675.

[194] E. Plenge, D. H. J. Poot, M. Bernsen, G. Kotek,G. Houston, P. Wielopolski, L. van der Weerd, W. J.Niessen, E. Meijering, Super-resolution methods inmri: can they improve the trade-off between reso-lution, signal-to-noise ratio, and acquisition time?,Magnetic resonance in medicine 68 (2012) 1983–1993.

[195] K. Bahrami, F. Shi, I. Rekik, Y. Gao, D. Shen, 7T-guided super-resolution of 3T MRI, Medical physics44 (2017) 1661–1677.

[196] G. Van Steenkiste, D. H. J. Poot, B. Jeuris-sen, A. J. den Dekker, F. Vanhevel, P. M.Parizel, J. Sijbers, Super-resolution t,javax.xml.bind.jaxbelement@1458c115, es-timation: Quantitative high resolutiont, javax.xml.bind.jaxbelement@62365375,mapping from a set of low resolution t,javax.xml.bind.jaxbelement@20656587, -weighted

36

images with different slice orientations., Magneticresonance in medicine 77 (2017) 1818–1830.

[197] K. Zeng, H. Zheng, C. Cai, Y. Yang, K. Zhang,Z. Chen, Simultaneous single- and multi-contrastsuper-resolution for brain MRI images based on aconvolutional neural network., Computers in biol-ogy and medicine 99 (2018) 133–141.

[198] C. Liu, X. Wu, X. Yu, Y. Tang, J. Zhang, J. Zhou,Fusing multi-scale information in convolution net-work for mr image super-resolution reconstruction.,Biomedical engineering online 17 (2018) 114.

[199] A. S. Chaudhari, Z. Fang, F. Kogan, J. Wood, K. J.Stevens, E. K. Gibbons, J. H. Lee, G. E. Gold,B. A. Hargreaves, Super-resolution musculoskele-tal MRI using deep learning, Magnetic resonancein medicine 80 (2018) 2139–2154.

[200] A. Jog, A. Carass, S. Roy, D. L. Pham, J. L. Prince,Random forest regression for magnetic resonanceimage synthesis, Medical image analysis 35 (2017)475–488.

[201] K. E. Keenan, M. Ainslie, A. J. Barker, M. A. Boss,K. M. Cecil, C. Charles, T. L. Chenevert, L. Clarke,J. L. Evelhoch, P. Finn, D. Gembris, J. L. Gunter,D. L. G. Hill, C. R. Jack, E. F. Jackson, G. Liu,S. E. Russek, S. D. Sharma, M. Steckner, K. F.Stupic, J. D. Trzasko, C. Yuan, J. Zheng, Quan-titative magnetic resonance imaging phantoms: Areview and the need for a system phantom, Mag-netic resonance in medicine 79 (2018) 48–61.

[202] K. Jurczuk, M. Kretowski, P.-A. Eliat, H. Saint-Jalmes, J. Bezy-Wendling, In silico modeling ofmagnetic resonance flow imaging in complex vascu-lar networks, IEEE transactions on medical imaging33 (2014) 2191–2209.

[203] Y. Zhou, S. Giffard-Roisin, M. De Craene,S. Camarasu-Pop, J. D’Hooge, M. Alessandrini,D. Friboulet, M. Sermesant, O. Bernard, A frame-work for the generation of realistic synthetic car-diac ultrasound and magnetic resonance imagingsequences from the same virtual patients., IEEEtransactions on medical imaging 37 (2018) 741–754.

[204] N. Duchateau, M. Sermesant, H. Delingette, N. Ay-ache, Model-based generation of large databases ofcardiac images: Synthesis of pathological cine MRsequences from real healthy cases, IEEE transac-tions on medical imaging 37 (2018) 755–766.

[205] A. Creswell, T. White, V. Dumoulin, K. Arulku-maran, B. Sengupta, A. A. Bharath, Generativeadversarial networks: An overview, IEEE SignalProcessing Magazine 35 (2018) 53–65.

[206] Y. Hong, U. Hwang, J. Yoo, S. Yoon, Howgenerative adversarial networks and their variantswork: An overview of GAN, arXiv preprintarXiv:1711.05914v7 (2017).

[207] H. Huang, P. S. Yu, C. Wang, An introductionto image synthesis with generative adversarial nets,arXiv preprint arXiv:1803.04469v1 (2018).

[208] A. Osokin, A. Chessel, R. E. C. Salas, F. Vaggi,Gans for biological image synthesis, in: Proc. IEEEInt. Conf. Computer Vision (ICCV), pp. 2252–2261.

[209] G. Antipov, M. Baccouche, J. Dugelay, Face agingwith conditional generative adversarial networks,in: Proc. IEEE Int. Conf. Image Processing (ICIP),pp. 2089–2093.

[210] C. Bodnar, Text to image synthesis using gen-erative adversarial networks, arXiv preprentarXiv:1805.00676v1 (2018).

[211] H. Dong, S. Yu, C. Wu, Y. Guo, Semantic imagesynthesis via adversarial learning, arXiv preprintarXiv:1707.06873v1 (2017).

[212] S. Reed, Z. Akata, X. Yan, L. Logeswaran,B. Schiele, H. Lee, Generative adversarial text toimage synthesis, arXiv preprint arXiv:1605.05396v2(2016).

[213] H.-C. Shin, N. A. Tenenholtz, J. K. Rogers, C. G.Schwarz, M. L. Senjem, J. L. Gunter, K. P. Andri-ole, M. Michalski, Medical Image Synthesis for DataAugmentation and Anonymization Using Genera-tive Adversarial Networks, in: International Work-shop on Simulation and Synthesis in Medical Imag-ing, Springer, pp. 1–11.

[214] T. C. W. Mok, A. C. S. Chung, Learningdata augmentation for brain tumor segmentationwith coarse-to-fine generative adversarial networks,arXiv 1805.11291 (2018).

[215] J. T. Guibas, T. S. Virdi, P. S. Li, Synthetic medicalimages from dual generative adversarial networks,arXiv preprint arXiv 1709.01872 (2017).

[216] A. Kitchen, J. Seah, Deep generative adversarialneural networks for realistic prostate lesion MRIsynthesis, arXiv preprint arXiv:1708.00129 (2017).

[217] D. Nie, R. Trullo, J. Lian, C. Petitjean, S. Ruan,Q. Wang, D. Shen, Medical image synthesiswith context-aware generative adversarial networks,Medical image computing and computer-assistedintervention : MICCAI ... International Confer-ence on Medical Image Computing and Computer-Assisted Intervention 10435 (2017) 417–425.

37

[218] K. D. Spuhler, J. Gardus, Y. Gao, C. DeLorenzo,R. Parsey, C. Huang, Synthesis of patient-specifictransmission image for PET attenuation correctionfor PET/MR imaging of the brain using a convolu-tional neural network, Journal of nuclear medicine(2018).

[219] A. Torrado-Carvajal, J. Vera-Olmos, D. Izquierdo-Garcia, O. A. Catalano, M. A. Morales, J. Margolin,A. Soricelli, M. Salvatore, N. Malpica, C. Catana,Dixon-VIBE deep learning (DIVIDE) pseudo-CTsynthesis for pelvis PET/MR attenuation correc-tion, Journal of nuclear medicine (2018).

[220] Q. Zhang, H. Wang, H. Lu, D. Won, S. W. Yoon,Medical image synthesis with generative adversarialnetworks for tissue recognition, in: Proc. IEEE Int.Conf. Healthcare Informatics (ICHI), pp. 199–207.

[221] M. Frid-Adar, E. Klang, M. Amitai, J. Goldberger,H. Greenspan, Synthetic data augmentation usingGAN for improved liver lesion classification, in:Proc. IEEE 15th Int. Symp. Biomedical Imaging(ISBI 2018), pp. 289–293.

[222] J. M. Wolterink, A. M. Dinkla, M. H. F. Savenije,P. R. Seevinck, C. A. T. van den Berg, I. Isgum,Deep MR to CT synthesis using unpaired data,arXiv:1708.01155v1 (2017).

[223] J. M. F. Calvin R. Maurer, Jr., A review of medicalimage registration, 1993.

[224] J. Maclaren, M. Herbst, O. Speck, M. Zaitsev,Prospective motion correction in brain imaging: areview, Magnetic resonance in medicine 69 (2013)621–636.

[225] M. Zaitsev, B. Akin, P. LeVan, B. R. Knowles,Prospective motion correction in functional MRI,Neuroimage 154 (2017) 33–42.

[226] O. Fluck, C. Vetter, W. Wein, A. Kamen, B. Preim,R. Westermann, A survey of medical image regis-tration on graphics hardware., Computer methodsand programs in biomedicine 104 (2011) e45–e57.

[227] L. Shi, W. Liu, H. Zhang, Y. Xie, D. Wang, A sur-vey of GPU-based medical image computing tech-niques., Quantitative imaging in medicine andsurgery 2 (2012) 188–206.

[228] A. Eklund, P. Dufort, D. Forsberg, S. M. LaConte,Medical image processing on the gpu - past, presentand future., Medical image analysis 17 (2013) 1073–1094.

[229] J. B. Maintz, M. A. Viergever, A survey of med-ical image registration., Medical image analysis 2(1998) 1–36.

[230] B. Glocker, A. Sotiras, N. Komodakis, N. Paragios,Deformable medical image registration: setting thestate of the art with discrete methods., Annual re-view of biomedical engineering 13 (2011) 219–244.

[231] A. Sotiras, C. Davatzikos, N. Paragios, Deformablemedical image registration: a survey., IEEE trans-actions on medical imaging 32 (2013) 1153–1190.

[232] F. P. M. Oliveira, J. M. R. S. Tavares, Medical im-age registration: a review., Computer methods inbiomechanics and biomedical engineering 17 (2014)73–93.

[233] P. K. Saha, R. Strand, G. Borgefors, Digital topol-ogy and geometry in medical imaging: A survey,IEEE transactions on medical imaging 34 (2015)1940–1964.

[234] M. A. Viergever, J. B. A. Maintz, S. Klein, K. Mur-phy, M. Staring, J. P. W. Pluim, A survey of medi-cal image registration - under review., Medical im-age analysis 33 (2016) 140–144.

[235] G. Song, J. Han, Y. Zhao, Z. Wang, H. Du, A re-view on medical image registration as an optimiza-tion problem., Current medical imaging reviews 13(2017) 274–283.

[236] E. Ferrante, N. Paragios, Slice-to-volume medicalimage registration: A survey., Medical image anal-ysis 39 (2017) 101–123.

[237] A. P. Keszei, B. Berkels, T. M. Deserno, Survey ofnon-rigid registration tools in medicine., Journal ofdigital imaging 30 (2017) 102–116.

[238] S. Nag, Image registration techniques: A survey,arXiv preprint arXiv:1712.07540v1 (2017).

[239] J. Jiang, P. Trundle, J. Ren, Medical image anal-ysis with artificial neural networks., Computerizedmedical imaging and graphics : the official journalof the Computerized Medical Imaging Society 34(2010) 617–631.

[240] G. Wu, M. Kim, Q. Wang, B. C. Munsell,D. Shen, Scalable high-performance image registra-tion framework by unsupervised deep feature rep-resentations learning., IEEE transactions on bio-medical engineering 63 (2016) 1505–1516.

[241] S. S. M. Salehi, S. Khan, D. Erdogmus,A. Gholipour, Real-time deep pose estimation withgeodesic loss for image-to-template rigid registra-tion., IEEE transactions on medical imaging (2018).

[242] D. Toth, S. Miao, T. Kurzendorfer, C. A. Rinaldi,R. Liao, T. Mansi, K. Rhode, P. Mountney, 3D/2Dmodel-to-image registration by imitation learningfor cardiac procedures., International journal ofcomputer assisted radiology and surgery (2018).

38

[243] X. Han, Mr-based synthetic CT generation using adeep convolutional neural network method, Medicalphysics 44 (2017) 1408–1419.

[244] M. Liu, D. Cheng, K. Wang, Y. Wang, A. D. N.Initiative, Multi-modality cascaded convolutionalneural networks for alzheimer’s disease diagnosis.,Neuroinformatics 16 (2018) 295–308.

[245] L. Xiang, Y. Qiao, D. Nie, L. An, Q. Wang, D. Shen,Deep auto-context convolutional neural networksfor standard-dose PET image estimation from low-dose PET/MRI., Neurocomputing 267 (2017) 406–416.

[246] S. Shan, W. Yan, X. Guo, E. I.-C. Chang, Y. Fan,Y. Xu, Unsupervised end-to-end learning for de-formable medical image registration, arXiv preprintarXiv:1711.08608v2 (2017).

[247] G. Balakrishnan, A. Zhao, M. R. Sabuncu, J. Gut-tag, A. V. Dalca, An unsupervised learning modelfor deformable medical image registration, arXivpreprint arXiv:1802.02604v3 (2018).

[248] B. D. de Vos, F. F. Berendsen, M. A. Viergever,H. Sokooti, M. Staring, I. Isgum, A deeplearning framework for unsupervised affine anddeformable image registration, arXiv preprintarXiv:1809.06130v1 (2018).

[249] M. W. Vannier, R. L. Butterfield, D. Jordan, W. A.Murphy, R. G. Levitt, M. Gado, Multispectral anal-ysis of magnetic resonance images., Radiology 154(1985) 221–224.

[250] A. Lundervold, K. Moen, T. Taxt, Automatic recog-nition of normal and pathological tissue types inMR images, in: Proc. of the NOBIM Conference,Oslo, Norway, 1988.

[251] T. Taxt, A. Lundervold, B. Fuglaas, H. Lien,V. Abeler, Multispectral analysis of uterine corpustumors in magnetic resonance imaging., Magneticresonance in medicine 23 (1992) 55–76.

[252] T. Taxt, A. Lundervold, Multispectral analysis ofthe brain using magnetic resonance imaging., IEEEtransactions on medical imaging 13 (1994) 470–481.

[253] A. Lundervold, G. Storvik, Segmentation of brainparenchyma and cerebrospinal fluid in multispectralmagnetic resonance images, IEEE Transactions onMedical Imaging 14 (1995) 339–349.

[254] M. Cabezas, A. Oliver, X. Llad, J. Freixenet, M. B.Cuadra, A review of atlas-based segmentation formagnetic resonance brain images, Computer meth-ods and programs in biomedicine 104 (2011) e158–e177.

[255] D. Garca-Lorenzo, S. Francis, S. Narayanan, D. L.Arnold, D. L. Collins, Review of automatic segmen-tation methods of multiple sclerosis white matter le-sions on conventional magnetic resonance imaging,Medical image analysis 17 (2013) 1–18.

[256] E. Smistad, T. L. Falch, M. Bozorgi, A. C. Elster,F. Lindseth, Medical image segmentation on GPUs–a comprehensive review., Medical image analysis 20(2015) 1–18.

[257] J. Bernal, K. Kushibar, D. S. Asfaw, S. Valverde,A. Oliver, R. Mart, X. Llad, Deep convolutionalneural networks for brain image analysis on mag-netic resonance imaging: a review, arXiv preprintarXiv:1712.03747v3 (2017).

[258] L. Dora, S. Agrawal, R. Panda, A. Abraham, State-of-the-art methods for brain tissue segmentation: Areview, IEEE Reviews in Biomedical Engineering 10(2017) 235–249.

[259] H. R. Torres, S. Queiros, P. Morais, B. Oliveira,J. C. Fonseca, J. L. Vilaa, Kidney segmentation inultrasound, magnetic resonance and computed to-mography images: A systematic review, Computermethods and programs in biomedicine 157 (2018)49–67.

[260] J. Bernal, K. Kushibar, D. S. Asfaw, S. Valverde,A. Oliver, R. Mart, X. Llad, Deep convolutionalneural networks for brain image analysis on mag-netic resonance imaging: a review, Artificial intel-ligence in medicine (2018).

[261] S. Moccia, E. De Momi, S. El Hadji, L. S. Mattos,Blood vessel segmentation algorithms - review ofmethods, datasets and evaluation metrics, Com-puter methods and programs in biomedicine 158(2018) 71–91.

[262] A. Makropoulos, S. J. Counsell, D. Rueckert, Areview on automatic fetal and neonatal brain MRIsegmentation., NeuroImage 170 (2018) 231–248.

[263] L. Chen, P. Bentley, D. Rueckert, Fully automaticacute ischemic lesion segmentation in DWI usingconvolutional neural networks, NeuroImage. Clini-cal 15 (2017) 633–643.

[264] M. Havaei, A. Davy, D. Warde-Farley, A. Biard,A. Courville, Y. Bengio, C. Pal, P.-M. Jodoin,H. Larochelle, Brain tumor segmentation with deepneural networks, Medical image analysis 35 (2017)18–31.

[265] H. Choi, K. H. Jin, Fast and robust segmentationof the striatum using deep convolutional neural net-works, Journal of Neuroscience Methods 274 (2016)146–153.

39

[266] B. Ibragimov, L. Xing, Segmentation of organs-at-risks in head and neck CT images using convolu-tional neural networks, Medical physics 44 (2017)547–557.

[267] T. L. Kline, P. Korfiatis, M. E. Edwards, J. D. Blais,F. S. Czerwiec, P. C. Harris, B. F. King, V. E. Tor-res, B. J. Erickson, Performance of an artificialmulti-observer deep neural network for fully auto-mated segmentation of polycystic kidneys., Journalof digital imaging 30 (2017) 442–448.

[268] Y. Guo, Y. Gao, D. Shen, Deformable mr prostatesegmentation via deep feature learning and sparsepatch matching., IEEE transactions on medicalimaging 35 (2016) 1077–1089.

[269] X. Li, Q. Dou, H. Chen, C.-W. Fu, X. Qi, D. L.Belav, G. Armbrecht, D. Felsenberg, G. Zheng, P.-A. Heng, 3D multi-scale FCN with random modal-ity voxel dropout learning for intervertebral disclocalization and segmentation from multi-modalityMR images, Medical image analysis 45 (2018) 41–54.

[270] J. Kleesiek, G. Urban, A. Hubert, D. Schwarz,K. Maier-Hein, M. Bendszus, A. Biller, Deep MRIbrain extraction: A 3D convolutional neural net-work for skull stripping, Neuroimage 129 (2016)460–469.

[271] H. Li, N. A. Parikh, L. He, A novel transfer learningapproach to enhance deep neural network classifica-tion of brain functional connectomes., Frontiers inneuroscience 12 (2018) 491.

[272] L.-L. Zeng, H. Wang, P. Hu, B. Yang, W. Pu,H. Shen, X. Chen, Z. Liu, H. Yin, Q. Tan,K. Wang, D. Hu, Multi-site diagnostic classificationof schizophrenia using discriminant deep learningwith functional connectivity MRI., EBioMedicine30 (2018) 74–85.

[273] J. Wasserthal, P. Neher, K. H. Maier-Hein, Tract-Seg - fast and accurate white matter tract segmen-tation, Neuroimage 183 (2018) 239–253.

[274] J. H. Cole, R. P. K. Poudel, D. Tsagkrasoulis,M. W. A. Caan, C. Steves, T. D. Spector, G. Mon-tana, Predicting brain age with deep learning fromraw imaging data results in a reliable and heritablebiomarker, Neuroimage 163 (2017) 115–124.

[275] M. Liu, J. Zhang, E. Adeli, D. Shen, Landmark-based deep multi-instance learning for brain diseasediagnosis., Medical image analysis 43 (2018) 157–168.

[276] J. Islam, Y. Zhang, Brain mri analysis forAlzheimer’s disease diagnosis using an ensemble sys-tem of deep convolutional neural networks, Braininformatics 5 (2018) 2.

[277] D. Lu, K. Popuri, G. W. Ding, R. Balachandar, Beg,M. Faisal, Multimodal and multiscale deep neuralnetworks for the early diagnosis of alzheimer’s dis-ease using structural MR and FDG-PET images,Scientific reports 8 (2018) 5697.

[278] P. Moeskops, J. de Bresser, H. J. Kuijf, A. M. Men-drik, G. J. Biessels, J. P. W. Pluim, I. Igum, Evalu-ation of a deep learning approach for the segmenta-tion of brain tissues and white matter hyperinten-sities of presumed vascular origin inmri., NeuroIm-age. Clinical 17 (2018) 251–262.

[279] R. Pizarro, H.-E. Assemlal, D. De Nigris, C. El-liott, S. Antel, D. Arnold, A. Shmuel, Using deeplearning algorithms to automatically identify thebrain mri contrast: Implications for managing largedatabases., Neuroinformatics (2018).

[280] K. R. Laukamp, F. Thiele, G. Shakirin, D. Zopfs,A. Faymonville, M. Timmer, D. Maintz,M. Perkuhn, J. Borggrefe, Fully automateddetection and segmentation of meningiomas usingdeep learning on routine multiparametric MRI,European radiology (2018).

[281] M. Perkuhn, P. Stavrinou, F. Thiele, G. Shakirin,M. Mohan, D. Garmpis, C. Kabbasch, J. Borggrefe,Clinical evaluation of a multiparametric deep learn-ing model for glioblastoma segmentation using het-erogeneous magnetic resonance imaging data fromclinical routine., Investigative radiology (2018).

[282] E. A. AlBadawy, A. Saha, M. A. Mazurowski, Deeplearning for segmentation of brain tumors: Impactof cross-institutional training and testing., Medicalphysics 45 (2018) 1150–1158.

[283] S. Cui, L. Mao, J. Jiang, C. Liu, S. Xiong, Au-tomatic semantic segmentation of brain gliomasfrom mri images using a deep cascaded neuralnetwork., Journal of healthcare engineering 2018(2018) 4940593.

[284] F. Hoseini, A. Shahbahrami, P. Bayat, Adaptaheadoptimization algorithm for learning deep cnn ap-plied to mri segmentation., Journal of digital imag-ing (2018).

[285] Y. Yoo, L. Y. W. Tang, T. Brosch, D. K. B. Li,S. Kolind, I. Vavasour, A. Rauscher, A. L. MacKay,A. Traboulsee, R. C. Tam, Deep learning of jointmyelin and T1w MRI features in normal-appearingbrain tissue to distinguish between multiplesclero-sispatients and healthy controls., NeuroImage. Clin-ical 17 (2018) 169–178.

[286] M. F. Bobo, S. Bao, Y. Huo, Y. Yao, J. Virostko,A. J. Plassard, I. Lyu, A. Assad, R. G. Abramson,M. A. Hilmes, B. A. Landman, Fully convolutional

40

neural networks improve abdominal organ segmen-tation., Proceedings of SPIE–the International So-ciety for Optical Engineering 10574 (2018).

[287] M. Shehata, F. Khalifa, A. Soliman, M. Ghazal,F. Taher, M. Abou El-Ghar, A. Dwyer,G. Gimel’farb, R. Keynton, A. El-Baz, Computer-aided diagnostic system for early detection of acuterenal transplant rejection using diffusion-weightedMRI, IEEE transactions on bio-medical engineering(2018).

[288] R. Cheng, H. R. Roth, N. Lay, L. Lu, B. Turk-bey, W. Gandler, E. S. McCreedy, T. Pohida, P. A.Pinto, P. Choyke, M. J. McAuliffe, R. M. Summers,Automatic magnetic resonance prostate segmenta-tion by deep learning with holistically nested net-works, Journal of medical imaging 4 (2017) 041302.

[289] J. Ishioka, Y. Matsuoka, S. Uehara, Y. Yasuda,T. Kijima, S. Yoshida, M. Yokoyama, K. Saito,K. Kihara, N. Numao, T. Kimura, K. Kudo, I. Ku-mazawa, Y. Fujii, Computer-aided diagnosis ofprostate cancer on magnetic resonance imaging us-ing a convolutional neural network algorithm, BJUinternational (2018).

[290] Y. Song, Y.-D. Zhang, X. Yan, H. Liu, M. Zhou,B. Hu, G. Yang, Computer-aided diagnosis ofprostate cancer using a deep convolutional neuralnetwork from multiparametric MRI, Journal ofmagnetic resonance imaging : JMRI (2018).

[291] X. Wang, W. Yang, J. Weinreb, J. Han, Q. Li,X. Kong, Y. Yan, Z. Ke, B. Luo, T. Liu, L. Wang,Searching for prostate cancer by fully automatedmagnetic resonance imaging classification: deeplearning versus non-deep learning., Scientific re-ports 7 (2017) 15415.

[292] X. Yang, C. Liu, Z. Wang, J. Yang, H. L. Min,L. Wang, K.-T. T. Cheng, Co-trained convolu-tional neural networks for automated detection ofprostate cancer in multi-parametric MRI, Medicalimage analysis 42 (2017) 212–227.

[293] M. H. Le, J. Chen, L. Wang, Z. Wang, W. Liu,K.-T. T. Cheng, X. Yang, Automated diagnosis ofprostate cancer in multi-parametric MRI based onmultimodal convolutional neural networks, Physicsin medicine and biology 62 (2017) 6497–6514.

[294] D. Forsberg, E. Sjoblom, J. L. Sunshine, Detectionand labeling of vertebrae in MR images using deeplearning with clinical annotations as training data,Journal of digital imaging 30 (2017) 406–412.

[295] J.-T. Lu, S. Pedemonte, B. Bizzo, S. Doyle, K. P.Andriole, M. H. Michalski, R. G. Gonzalez, S. R.Pomerantz, DeepSPINE: automated lumbar verte-bral segmentation, disc-level designation, and spinal

stenosis grading using deep learning, arXiv preprintarXiv:1807.10215v1 (2018).

[296] Z. Han, B. Wei, S. Leung, I. B. Nachum, D. Lai-dley, S. Li, Automated pathogenesis-based diag-nosis of lumbar neural foraminal stenosis via deepmultiscale multitask learning., Neuroinformatics 16(2018) 325–337.

[297] K. H. Kim, W.-J. Do, S.-H. Park, Improving res-olution of MR images with an adversarial networkincorporating images with different contrast, Med-ical physics 45 (2018) 3120–3131.

[298] A. H. Pilevar, CBMIR: content-based image re-trieval algorithm for medical image databases, Jour-nal of medical signals and sensors 1 (2011) 12–18.

[299] A. Kumar, J. Kim, W. Cai, M. Fulham, D. Feng,Content-based medical image retrieval: a survey ofapplications to multidimensional and multimodal-ity dat., Journal of digital imaging 26 (2013) 1025–1039.

[300] A. V. Faria, K. Oishi, S. Yoshida, A. Hillis,M. I. Miller, S. Mori, Content-based image re-trieval for brain mri: an image-searching engineand population-based analysis to utilize past clin-ical data for future diagnosis, NeuroImage. Clinical7 (2015) 367–376.

[301] A. Kumar, F. Nette, K. Klein, M. Fulham, J. Kim,A visual analytics approach using the exploration ofmultidimensional feature spaces for content-basedmedical image retrieval., IEEE journal of biomedi-cal and health informatics 19 (2015) 1734–1746.

[302] M. V. N. Bedo, D. Pereira Dos Santos, M. Ponciano-Silva, P. M. de Azevedo-Marques, A. P. d. L. Fer-reira de Carvalho, C. Traina, Endowing a content-based medical image retrieval system with percep-tual similarity using ensemble strategy, Journal ofdigital imaging 29 (2016) 22–37.

[303] C. Muramatsu, Overview on subjective similarityof images for content-based medical image retrieval,Radiological physics and technology (2018).

[304] A. B. Spanier, N. Caplan, J. Sosna, B. Acar,L. Joskowicz, A fully automatic end-to-end methodfor content-based image retrieval of CT scans withsimilar liver lesion annotations, International jour-nal of computer assisted radiology and surgery 13(2018) 165–174.

[305] A. Gordo, J. Almazan, J. Revaud, D. Larlus, End-to-end learning of deep visual representations forimage retrieval (????).

[306] P. Liu, J. Guo, C. Wu, D. Cai, Fusion of deep learn-ing and compressed domain features for content-based image retrieval, IEEE Transactions on ImageProcessing 26 (2017) 5706–5717.

41

[307] J. Han, D. Zhang, G. Cheng, N. Liu, D. Xu,Advanced deep-learning techniques for salient andcategory-specific object detection: A survey, IEEESignal Processing Magazine 35 (2018) 84–100.

[308] T. Piplani, D. Bamman, Deepseek: Contentbased image search & retrieval, arXiv preprintarXiv:1801.03406v2 (2018).

[309] J. Yang, J. Liang, H. Shen, K. Wang, P. L. Rosin,M. Yang, Dynamic match kernel with deep convo-lutional features for image retrieval, IEEE Transac-tions on Image Processing 27 (2018) 5288–5302.

[310] J. E. S. Sklan, A. J. Plassard, D. Fabbri, B. A.Landman, Toward content based image retrievalwith deep convolutional neural networks, Proceed-ings of SPIE–the International Society for OpticalEngineering 9417 (2015).

[311] R. S. Bressan, D. H. A. Alves, L. M. Valerio, P. H.Bugatti, P. T. M. Saito, DOCToR: the role of deepfeatures in content-based mammographic image re-trieval, in: Proc. IEEE 31st Int. Symp. Computer-Based Medical Systems (CBMS), pp. 158–163.

[312] A. Qayyum, S. M. Anwar, M. Awais, M. Majid,Medical image retrieval using deep convolutionalneural network, arXiv preprint arXiv:1703.08472v1(2017).

[313] Y.-A. Chung, W.-H. Weng, Learning deep represen-tations of medical images using siamese CNNs withapplication to content-based image retrieval, arXivpreprint arXiv:1711.08490v2 (2017).

[314] B. Jing, P. Xie, E. Xing, On the automatic gen-eration of medical imaging reports, arXiv preprintarXiv:1711.08195v3 (2017).

[315] C. Y. Li, X. Liang, Z. Hu, E. P. Xing, Hy-brid retrieval-generation reinforced agent for med-ical image report generation, arXiv preprintarXiv:1805.08298v1 (2018).

[316] M. Moradi, A. Madani, Y. Gur, Y. Guo, T. Syeda-Mahmood, Bimodal network architectures for au-tomatic generation of image annotation from text,arXiv preprint arXiv:1809.01610v1 (2018).

[317] Y. Zhang, D. Y. Ding, T. Qian, C. D. Manning,C. P. Langlotz, Learning to summarize radiologyfindings, arXiv preprint arXiv:1809.04698v1 (????).

[318] E. Pons, L. M. M. Braun, M. G. M. Hunink, J. A.Kors, Natural language processing in radiology: Asystematic review, Radiology 279 (2016) 329–343.

[319] J. Zech, M. Pain, J. Titano, M. Badgeley, J. Schef-flein, A. Su, A. Costa, J. Bederson, J. Lehar, E. K.Oermann, Natural language-based machine learn-ing models for the annotation of clinical radiologyreports, Radiology 287 (2018) 570–580.

[320] D. J. Goff, T. W. Loehfelm, Automated radiologyreport summarization using an open-source natu-ral language processing pipeline, Journal of digitalimaging 31 (2018) 185–192.

[321] E. Gibson, W. Li, C. Sudre, L. Fidon, D. I. Shakir,G. Wang, Z. Eaton-Rosen, R. Gray, T. Doel, Y. Hu,T. Whyntie, P. Nachev, M. Modat, D. C. Bar-ratt, S. Ourselin, M. J. Cardoso, T. Vercauteren,NiftyNet: a deep-learning platform for medicalimaging, Computer methods and programs inbiomedicine 158 (2018) 113–122.

[322] W. Li, G. Wang, L. Fidon, S. Ourselin, M. J. Car-doso, T. Vercauteren, On the compactness, effi-ciency, and representation of 3d convolutional net-works: Brain parcellation as a pretext task, in: In-ternational Conference on Information Processingin Medical Imaging (IPMI).

[323] K. Kamnitsas, C. Ledig, V. F. Newcombe, J. P.Simpson, A. D. Kane, D. K. Menon, D. Rueckert,B. Glocker, Efficient multi-scale 3D CNN with fullyconnected CRF for accurate brain lesion segmenta-tion, Medical image analysis 36 (2017) 61–78.

[324] O. Ronneberger, P.Fischer, T. Brox, U-net: Con-volutional networks for biomedical image segmenta-tion, in: Medical Image Computing and Computer-Assisted Intervention (MICCAI), volume 9351 ofLNCS, Springer, 2015, pp. 234–241. (available onarXiv:1505.04597 [cs.CV]).

[325] V. Badrinarayanan, A. Kendall, R. Cipolla, Seg-Net: A Deep Convolutional Encoder-Decoder Ar-chitecture for Image Segmentation, IEEE Transac-tions on Pattern Analysis and Machine Intelligence(2017).

[326] M. Mardani, E. Gong, J. Y. Cheng, S. Vasanawala,G. Zaharchuk, M. Alley, N. Thakur, S. Han,W. Dally, J. M. Pauly, et al., Deep generative adver-sarial networks for compressed sensing automatesmri, arXiv preprint arXiv:1706.00051 (2017).

[327] S. Parisot, S. I. Ktena, E. Ferrante, M. Lee, R. G.Moreno, B. Glocker, D. Rueckert, Spectral graphconvolutions for population-based disease predic-tion, in: International Conference on Medical Im-age Computing and Computer-Assisted Interven-tion, Springer, pp. 177–185.

[328] G. Marcus, Deep learning: A critical appraisal,arXiv preprint arXiv:1801.00631 (2018).

[329] Z. C. Lipton, J. Steinhardt, Troubling Trends inMachine Learning Scholarship (2018).

[330] C. Zhang, S. Bengio, M. Hardt, B. Recht,O. Vinyals, Understanding deep learning re-quires rethinking generalization, arXiv preprintarXiv:1611.03530 (2016).

42

[331] M. Fredrikson, S. Jha, T. Ristenpart, Model in-version attacks that exploit confidence informationand basic countermeasures, in: Proceedings of the22nd ACM SIGSAC Conference on Computer andCommunications Security, ACM, pp. 1322–1333.

[332] R. Shokri, M. Stronati, C. Song, V. Shmatikov,Membership inference attacks against machinelearning models, in: Security and Privacy (SP),2017 IEEE Symposium on, IEEE, pp. 3–18.

[333] B. McMahan, E. Moore, D. Ramage, S. Hampson,B. A. y Arcas, Communication-Efficient Learningof Deep Networks from Decentralized Data, in:A. Singh, J. Zhu (Eds.), Proceedings of the 20th

International Conference on Artificial Intelligenceand Statistics, volume 54 of Proceedings of MachineLearning Research, PMLR, Fort Lauderdale, FL,USA, 2017, pp. 1273–1282.

[334] O. Gupta, R. Raskar, Distributed learning of deepneural network over multiple agents, Journal of Net-work and Computer Applications 116 (2018) 1–8.

[335] P. Vepakomma, O. Gupta, T. Swedish, R. Raskar,Split learning for health: Distributed deep learningwithout sharing raw patient data, arXiv preprintarXiv:1812.00564 (2018).

[336] N. Papernot, M. Abadi, U. Erlingsson, I. Goodfel-low, K. Talwar, Semi-supervised knowledge transferfor deep learning from private training data, arXivpreprint arXiv:1610.05755 (2016).

[337] N. Papernot, S. Song, I. Mironov, A. Raghunathan,K. Talwar, U. Erlingsson, Scalable Private Learn-ing with PATE, arXiv preprint arXiv:1802.08908(2018).

[338] H. B. McMahan, D. Ramage, K. Talwar, L. Zhang,Learning Differentially Private Recurrent LanguageModels, in: International Conference on LearningRepresentations.

[339] P. Vepakomma, T. Swedish, R. Raskar, O. Gupta,A. Dubey, No Peek: A Survey of private distributeddeep learning, arXiv preprint arXiv:1812.03288(2018).

[340] K. Chang, N. Balachandar, C. Lam, D. Yi,J. Brown, A. Beers, B. Rosen, D. L. Rubin,J. Kalpathy-Cramer, Distributed deep learning net-works among institutions for medical imaging, Jour-nal of the American Medical Informatics Associa-tion : JAMIA 25 (2018) 945–954.

[341] J. R. Zech, M. A. Badgeley, M. Liu, A. B. Costa,J. J. Titano, E. K. Oermann, Variable generaliza-tion performance of a deep learning model to detectpneumonia in chest radiographs: A cross-sectionalstudy, PLoS medicine 15 (2018).

[342] A. Lundervold, A. Lundervold, J. Rørvik, Fast semi-supervised segmentation of the kidneys in DCE-MRI using convolutional neural networks and trans-fer learning, 2017.

[343] A. Lundervold, K. Sprawka, A. Lundervold, Fastestimation of kidney volumes and time coursesin DCE-MRI using convolutional neural networks,2018.

[344] G. E. Hinton, A. Krizhevsky, S. D. Wang, Trans-forming auto-encoders, in: International Confer-ence on Artificial Neural Networks, Springer, pp.44–51.

[345] S. Sabour, N. Frosst, G. E. Hinton, Dynamic rout-ing between capsules, in: Advances in Neural Infor-mation Processing Systems, pp. 3856–3866.

[346] V. Mnih, N. Heess, A. Graves, et al., Recurrentmodels of visual attention, in: Advances in neuralinformation processing systems, pp. 2204–2212.

[347] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville,R. Salakhudinov, R. Zemel, Y. Bengio, Show, at-tend and tell: Neural image caption generation withvisual attention, in: International conference onmachine learning, pp. 2048–2057.

[348] D. Castelvecchi, Can we open the black box of AI?,Nature News 538 (2016) 20.

[349] C. Olah, A. Satyanarayan, I. Johnson, S. Carter,L. Schubert, K. Ye, A. Mordvintsev, The buildingblocks of interpretability, Distill 3 (2018).

[350] G. Montavon, W. Samek, K.-R. Muller, Methodsfor interpreting and understanding deep neural net-works, Digital Signal Processing (2017).

[351] J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, H. Lip-son, Understanding neural networks through deepvisualization, in: Deep Learning Workshop, 31st In-ternational Conference on Machine Learning, 2015.

[352] C. Olah, A. Mordvintsev, L. Schubert, Feature vi-sualization, Distill 2 (2017).

[353] F. M. Hohman, M. Kahng, R. Pienta, D. H. Chau,Visual Analytics in Deep Learning: An Interroga-tive Survey for the Next Frontiers, IEEE Trans-actions on Visualization and Computer Graphics(2018).

[354] R. M. Neal, Bayesian learning for neural networks,Ph.D. thesis, University of Toronto, 1995.

[355] D. J. MacKay, A practical Bayesian framework forbackpropagation networks, Neural computation 4(1992) 448–472.

[356] P. Dayan, G. E. Hinton, R. M. Neal, R. S. Zemel,The Helmholtz machine, Neural computation 7(1995) 889–904.

43

[357] Y. Li, Y. Gal, Dropout Inference in Bayesian NeuralNetworks with Alpha-divergences, in: InternationalConference on Machine Learning, pp. 2052–2061.

[358] C. Leibig, V. Allken, M. S. Ayhan, P. Berens,S. Wahl, Leveraging uncertainty information fromdeep neural networks for disease detection, Scien-tific reports 7 (2017) 17816.

[359] A. Kendall, V. Badrinarayanan, R. Cipolla,Bayesian segnet: Model uncertainty in deep con-volutional encoder-decoder architectures for sceneunderstanding, arXiv preprint arXiv:1511.02680(2015).

[360] R. Feinman, R. R. Curtin, S. Shintre, A. B. Gard-ner, Detecting adversarial samples from artifacts,arXiv preprint arXiv:1703.00410 (2017).

[361] P. Sharp, S. Hockfield, Convergence: The future ofhealth, Science 355 (2017) 589.

[362] L. Hood, M. Flores, A personal view on sys-tems medicine and the emergence of proactive P4medicine: predictive, preventive, personalized andparticipatory, New biotechnology 29 (2012) 613–624.

44