Prospects of deep learning for medical imaging

CROSSMARK_logo_3_Test

1/1

REVIEW ARTICLE

Precision and Future Medicine [Epub ahead of print] http://doi.org/10.23838/pfm.2018.00030 pISSN: 2508-7940 · eISSN: 2508-7959

https://crossmark-cdn.crossref.org/widget/v2.0/logos/CROSSMARK_Color_square.svg 2017-03-16

Prospects of deep learning for medical imaging Jonghoon Kim1,2, Jisu Hong1,2, Hyunjin Park2,3

Department of Electronic, Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Korea

1

Center for Neuroscience Imaging Research, Institute for Basic Science (IBS), Suwon, Korea

2

School of Electronic and Electrical Engineering, Sungkyunkwan University, Suwon, Korea

3

Received: March 10, 2018 Revised: April 11, 2018 Accepted: April 14, 2018 Corresponding author: Hyunjin Park School of Electronic and Electrical Engineering, Sungkyunkwan University, 2066 Seobu-ro, Jangan-gu, Suwon 16419, Korea Tel: +82-31-299-4956 E-mail: [email protected]

ABSTRACT

Machine learning techniques are essential components of medical imaging research. Recently, a highly flexible machine learning approach known as deep learning has emerged as a disruptive technology to enhance the performance of existing machine learning techniques and to solve previously intractable problems. Medical imaging has been identified as one of the key research fields where deep learning can contribute significantly. This review article aims to survey deep learning literature in medical imaging and describe its potential for future medical imaging research. First, an overview of how traditional machine learning evolved to deep learning is provided. Second, a survey of the application of deep learning in medical imaging research is given. Third, wellknown software tools for deep learning are reviewed. Finally, conclusions with limitations and future directions of deep learning in medical imaging are provided. Keywords: Deep learning; Diagnostic imaging; Machine learning

INTRODUCTION

Machine learning techniques have been widely used in medical imaging research in the form of many successful classifier and clustering algorithms [1,2]. Many clinicians are well aware of the effectiveness of classifiers, such as support vector machine (SVM), and clustering algo-

rithms, such as k-nearest neighbor (k-NN) [3]. Recently, deep learning (DL) has emerged as the

go-to methodology to drastically enhance the performance of existing machine learning techniques and to solve previously intractable problems. In addition, DL is a generic methodology

that has a disruptive impact in other scientific fields as well. Thus, it has become imperative for medical imaging researchers to fully embrace DL technology. This review article is borne out of that necessity. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http:// creativecommons.org/licenses/ by-nc/4.0/).

Medical image processing refers to a set of procedures to obtain clinically meaningful infor-

mation from various imaging modalities, mostly for diagnosis or prognosis. The modalities are typically in vivo types, but ex vivo imaging could be used for medical image processing as well.

The extracted information could be used to enhance diagnosis and prognosis according to the

patient’s needs. Distinct medical imaging modalities, such as magnetic resonance imaging (MRI), computed tomography (CT), and positron emission tomography (PET), could provide

Copyright © 2018 Sungkyunkwan University School of Medicine

1

Deep learning for medical imaging distinct information for the patient being imaged. Structural

given problem, and many studies depend on trial-and-error

and these are used as quantitative features for future diag-

is choosing the right features to correctly model a given

and functional information could be extracted as necessary, nosis and prognosis. Research in medical image processing

typically aims to extract features that might be difficult to as-

to find the right combination. A major challenge in particular problem.

The development of an artificial neural network (ANN)

sess with the naked eye. There are two types of features. The

largely circumvents this problem by learning feature repre-

experts, and the other is the agonistic feature defined by

ture extraction procedure [10]. ANN attempts to mimic hu-

mathematical equations [4]. The agonistic features suffer

from less operator bias than the semantic features. Still, se-

mantic features are well recognized in radiology research, which is an accumulation of years of human expertise. How-

ever, many semantic features are time-consuming to compute and sometimes there are inconsistencies among ex-

perts. The extracted agonistic features might be used as imaging biomarkers to explain various states of the patient. A recent research approach known as the “radiomics” approach employs hundreds or thousands of agnostic features

to obtain clinically relevant information for diagnosis and prognosis [5,6].

Machine learning approaches are applied to associate im-

aging features obtained from medical image processing with

relevant clinical information. Machine learning started as a field in computer science to endow algorithms to solve prob-

lems without being explicitly programmed. It typically learns representations from training data, which are generalized in

separate test data. The technology has been applied to com-

puter-aided diagnosis (CADx) and computer-aided detection

(CADe) in medical imaging [7,8]. The CADx system can identify a disease-related region and quantify the properties of

that region, which could be used as a guidance for surgeons

and radiologists. Despite their usefulness in analyzing medical imaging, machine learning approaches have several lim-

itations. They show excellent performance when applied to training data but typically suffer losses in performances when applied to independent validation data [9]. This is partly due to the overfitting of the training data. Performance of machine learning techniques must be evaluated with both training and independent validation data. Many machine

learning studies have demonstrated great technical potential but only a few have shown actual clinical efficacy including

gains in survival. Machine learning techniques also have issues related to feature definition. For example, they rely on a pre-defined set of features. Additionally, sometimes the fea-

tures are difficult to define for a given problem. Researchers

need to choose from different combinations of features, algorithms, and degrees of complexity to sufficiently solve a 2

sentation directly from the raw input data, skipping the feaman brain processes using a network composed of intercon-

nected nodes. The initial ANN model was a simple feed-forward network known as perceptron that could perform lin-

ear classification tasks [11]. The early perceptron models re-

quired complex computation power beyond what was typically available at that time. Multilayer perceptron (MLP) was

proposed to improve the simple perceptron model by adding hidden layers and developing learning techniques, such as

back-propagation [12]. MLP formed the basis for the modern

DL approaches. The “deep” portion of DL refers to having many layers whose structures are suitable to model big data.

On a practical side, the DL approach requires a high computational load. With the recent developments of computation-

al infrastructure, such as graphical processing units (GPUs) and cloud computing systems, DL has become practical and

has achieved groundbreaking results in many fields. In par-

ticular, the visual recognition challenge based on large-scale data (ImageNet) performs the task of classifying among

1,000 objects leveraging 1,200,000 training and 100,000 test

images. Since the initiation of the challenge in 2010, the performance of DL algorithms gradually increased, and it began to exceed human accuracy beginning in 2015 (Fig. 1).

DL has been successfully applied to many research fields 30

ImageNet classification top-5 error (%)

first is the well-known semantic feature defined by human

25 20 15 10 5 0

28.2

25.8

16.4

11.7

7.3

6.7

5.1

3.5

ILSVRC ILSVRC ILSVRC ILSVRC ILSVRC ILSVRC Human ILSVRC 2010 2011 2012 2013 2014 2014 Performance 2015 NEC Xerox AlexNet Clarifi VGG GoogleNet ResNet America

Fig. 1. ImageNet classification performance in chronological order. ILSVRC, ImageNet Large Scale Visual Recognition Challenge.

http://pfmjournal.org

Jonghoon Kim, et al. and has become ubiquitous. As such, many studies have al-

the model associates input data with pre-defined categorical

increasingly more studies will adopt DL for medical imaging

crete categorical variable in classification. In regression, the

ready adopted DL to improve medical imaging research, and

research in the future. This review article aims to survey DL

literature in medical imaging and describe its potential for future medical imaging research. First, we provide an overview of how traditional machine learning evolved to DL. Second, we survey the application of DL in medical imaging research. Third, software tools for DL are reviewed. Finally, we

conclude with limitations and future directions of DL in medical imaging.

continuous variable in the regression. In un-supervised

learning, we use unlabeled input data to learn intrinsic pat-

terns in the input data. Un-supervised learning is commonly used in clustering. With clustering, we might identify sub-

groups within a given group of patients diagnosed with the same disease. Supervised learning is more costly to prepare, ANN is a statistical machine learning method inspired by

solve research problems [13]. There are two categories: supervised learning and un-supervised learning. In supervised learning, we train models using input data with matched la-

bels [14]. The model is a mathematical model that can associate input data with the matched labels and a predictive

model, which is validated using unseen test data. For exam-

ple, we start with MRI images labeled either as a normal con-

trol or diseased, and machine learning would lead to a mathematical model that could associate MRI images with diagnosis in both training and test data. Supervised learning is

commonly used in the following two tasks. In classification,

Input Weight

(i.e., the degree of symptoms) [18]. The output is typically a

requires human intervention.

Machine learning involves building data-driven models to

Nucleus

model associates input data with often continuous results

as it involves annotating input data with labels, which often

FROM TRADITIONAL MACHINE LEARNING TO DEEP LEARNING

Dendrites

results (i.e., normal vs. diseased) [15-17]. The output is a dis-

brain mechanism from neuroscience (Fig. 2) [19]. Research-

ers designed a learning algorithm that resembles how the brain handles information. A neuron is the basic unit of the

brain mechanism. The neuron is an electrically excitable cell that receives signals from other neurons, processes the re-

ceived information, and transmits electrical and chemical signals to other neurons. The input signal to a given neuron needs to exceed a certain threshold for it be activated and

further transmit a signal. The neurons are interconnected and form a network that collectively steers the brain mecha-

nism. ANN is an abstraction of an interconnected network of neurons with layers of nodes, and it consists of an input layer aggregating the input signal from other connected neurons,

a hidden layer responsible for training, and an output layer [20]. Each node takes the input from nodes from the previ-

Axon Input layer

Hidden layers

Output layer

Axon terminals

ous layer using various weights and computes the activation function, which is relayed onto the next layer of nodes. The

activation function approximates the complex process of a

physical neuron, which regulates the strength of the neuronal output in a non-linear manner. The mathematical pro-

cessing in a node can be represented using the following

A

equation: Output

Output= φ(W x+b). T

A node takes in an input value ‘x’ and multiplies it by weight

C B Fig. 2. Overview of artificial neural network. (A) In a biological neuron, the nucleus transforms the chemical inputs from the dendrites into electric signals. The signals are transmitted to the next neurons through the axon terminals. (B) In a node (perceptron), the input values are transformed by the weights, biases, and activation functions. The output values are transmitted to the next perceptron. (C) Multilayer perceptron consists of multiple perceptrons.

http://doi.org/10.23838/pfm.2018.00030

‘W,’ and then a bias of ‘b’ is added, which is fed to the activation function ‘φ.’ Differences between the output of ANN and the target value (i.e., ground truth) are called errors or losses. The training of the ANN is a procedure to update weights that

interconnect different nodes to explain the training data, which is equivalent to minimizing the loss value. The losses

are back-propagated through the network, and they are used to modify the weights and biases [21,22]. As a result, components of ANN are updated to best explain the target values.

3

Deep learning for medical imaging A deep neural network (DNN) is an expanded ANN incorpo-

many variants of DL architecture, and an exhaustive review

model [23-26]. DNN consists of an input layer, many hidden

mention three well-known DL architectures: convolutional

rating many hidden layers to increase the flexibility of the

of them is outside the scope this paper. In this paper, we

layers, and an output layer. The added layers afford solving

neural network (CNN), auto-encoder (AE), and recurrent neu-

more complex problems but can cause other issues. The first

ral network (RNN).

issue is the vanishing gradient problem [27]. This occurs because the input information cannot be effectively used to

Convolutional neural network

to travel the many layers of the activation function. This

stacked convolutional layers [28]. Convolution is a mathe-

CNN refers to a network architecture composed of many

update the weights of deep layers, as the information needs

matical operation based on two functions, addition and mul-

leads to under-training of the deep layers. Another issue is

tiplication. For a given input image, convolution is applied to

over-fitting, which occurs frequently when the complexity of

the input image based on a receptive filed (i.e., extents of

the model increases [25]. A complex model might explain the

convolution operation), which is analogous to the extents of

training data well, but it does not necessarily explain the un-

the response region of human vision. The convolution proce-

seen test data well. The increased complexity of many layers

dure is well-suited for image recognition because it considers

requires a high computational load and thus degrades the

locally connected information (i.e., neighboring voxels or pix-

training efficiency. The issues of vanishing gradient and

els). There are pooling layers between convolution layers.

over-fitting have been mitigated by enhanced activation

The pooling layer is essential to increasing the field of view of

function, cost function design, and drop-out approaches.

the network. It takes a portion of the locally connected nodes

The issue of the high computational load has been dealt with

of the input layer and results in an output that has a smaller

by using highly parallel hardware, such as GPUs and batch

spatial footprint. For example, we might take the maximum

normalization.

value of the outputs of four neighboring nodes. It can control

Modern DL approaches employ dedicated topology of

the over-fitting problems and reduce the computational re-

deeply stacked layers to solve a given problem. There are

Input layer

Kernel

Convolutional layer

A

Input layer

Max-pooling layer

B

Kernels

Label A Label B Label C Input data

Convolutional Pooling layer layer

Fully-connected layers

Output

C

Fig. 3. Overview of convolutional neural network (CNN). (A) The output of the convolutional layer is obtained by convolving its input layer and kernel weights. (B) The max-pooling layer outputs a down-sampled layer by extracting the maximum values from the partially windowed area of the input layer. (C) A general CNN consists of a series of convolutional layers, pooling layers, and fully connected layers.

4


Jonghoon Kim, et al. quirements. In general, the max-pooling algorithm is used

process, the hidden layer can represent its input layer with a

the receptive sliding window (Fig. 3).

tion. This model generally has an hourglass-shaped structure

for many architectures, extracting the highest values from As the last component of CNN, a series of fully connected

layers follow. The fully connected layer combines all activa-

tions of the previous layers. This layer works the same as in

reduced dimension that minimizes the loss of input informawith a dimension of the hidden layer being smaller than the input layer.

Stacked auto-encoder (SAE) is made by stacking the AEs so

the general ANN. After the series of fully connected layers,

that the output of an AE layer is treated as an input layer for

for the given problem. CNN was the initial catalyst for wide-

ture to obtain a denoised image of input data. The model

the model outputs the final set of feature values appropriate

spread adoption of DL. LeNet-5 is one of the early architectures that drastically improved existing machine learning approaches for recognizing hand-written digits [29], and there

another AE (Fig. 4) [35]. Vincent et al. [36] used this SAE strucused the greedy training method to learn several SAEs independently.

are other well-known architectures, including AlexNet, Goo-

Recurrent neural network

the architecture are found in the references.

tion starts from the input layer, travels through many hidden

gLeNet, VGG-Net, and ResNET [28,30-32]. Further details of

General ANN is a feed-forward network where input informa-

layers, and finally arrives at the output layer. RNN has a dif-

Auto-encoder and stacked auto-encoder

AE is a network that is used to derive an efficient representa-

tion of the input data [33,34]. As the name suggests, this network reduces the input layers to a layer with a small number of nodes (i.e., encoding) and then restores them via the upscaling process (i.e., decoding), as shown in Fig. 4. By this

ferent topology for its layer configuration. The output of hidden layers is fed back to an input layer using feedback (Fig. 5) [37,38]. RNN considers both current input data and feedback data from previous states and thus is well-suited for model-

ing sequential data, including both temporal and spatial information.

DEEP LEARNING APPLICATION IN MEDICAL IMAGING

In this section, we review representative applications of DL in medical imaging. First, CADx/CADe tasks are covered with Input Hidden Output layer layer layer Kernels

topics of classification, detection, and prediction. Second, A

Input layer

Hidden layers

Output layer

image processing tasks are covered with topics of segmentaB

tion, registration, and generation. CADx/CADe has many criti-

cal usages in radiology, such as finding an altered structural/

functional area (detection or localization), predicting the state of the object of interest based on a probabilistic model (prediction), and classifying binary or multiclass categories

(classification). In contrast, image processing is the prerequi-

site for subsequent clinical tasks, such as diagnosis or progIt+1 Input data

Hidden layers

Output (Input data’)

C

Fig. 4. Overview of auto-encoder (AE). (A) An AE consists of multiple perceptrons that mimic the input layer. (B) A stacked auto-encoder (SAE) can be built by stacking the AEs. (C) A general SAE is trained to reconstruct the output that is similar to the given input.

http://doi.org/10.23838/pfm.2018.00030

Ot

RNN cell

Ot+1

Input layer Recurrent = layer Output layer

I0 RNN cell

O0

O0

I1 RNN cell

O1

O1

I2 RNN cell

O2

O2

Fig. 5. A recurrent neural network (RNN) cell calculates the output by using its sequential inputs recurrently.

5

Deep learning for medical imaging nosis, and they can be categorized into the following tasks: a

often a pre-requisite for therapy planning. Many studies ad-

segmentation task assigns voxels with similar properties into

opted CNN and reported precise detection performance. Dou

onto a common spatial frame, and an enhancement task im-

lesion of cerebral microbleeds from MRI. Zreik et al. [40] pro-

labels, a registration task spatially aligns two or more images

proves the image quality for the given task. Each topic is de-

scribed in terms of the network architecture (e.g., CNN, RNN, or SAE) and its clinical application.

Applications using CNN architecture

CNN computes features from locally connected voxels (i.e., neighborhood voxels); hence, it is widely used as the reference model for medical image analysis. In general, a given

image goes through many convolution layers, and the fea-

tures are extracted from different layers. The extracted fea-

et al. [39] applied three-dimensional (3D) CNNs to localize the posed an automatic detection method for identifying left ventricle myocardium in coronary CT angiography using CNN. Using both PET/CT imaging, whole body multiple myeloma detection showed better performance than traditional machine

learning approaches, such as the random forest classifier, k-NNs, and SVM [41]. Apart from the common imaging modal-

ities (i.e., MRI, CT, and PET), retinal and endoscopy images, as well as histology images, were analyzed for finding the diseased region of interest (ROI) in other studies [17,42,43].

Prediction : The prediction task involves forecasting proper-

tures are further used for various tasks.

ties (often not evident to the naked eye) of an object using

CADx/CADe application using CNN architecture

dinal setting, where baseline imaging findings are used to

CADx/CADe applications based on CNN are shown in Table 1.

Detection: The detection task refers to finding the region as-

sociated with the diseased condition. The detection task is sometimes referred to as the localization task, as it involves

finding voxel positions associated with the disease. The task is

imaging analysis results. Prediction often occurs in a longitupredict properties of the object in the future. Many studies

attempted to compute the likelihood of clinical variables, such as drug or therapeutic response, patient’s survival, and disease grading, using imaging analysis results. Kawahara et

al. [44] proposed a novel CNN framework, BrainNetCNN, to

Table 1. CADx/CADe applications using convolutional neural network architecture Task

Detection

Modalities

Brain

Lesion detection [39]

PET/CT

Bone


CT

Endoscopic image Histology image MRI

MRI

PET

OCT MRI Classification

Clinical goal

MRI

Retina image

Prediction

Object

CT

CT

CT CT

Mammography RGB

Fluorescent image

Heart Retina

Stomach Breast Brain Liver

Esophagus Retina Brain Lung

Brain Lung Liver

Breast Skin Cell

Disease detection [40] Lesion detection [17]

Disease detection [42] Lesion detection [43]

Disease prediction [44] Disease prediction [45]

Treatment response prediction [46] Treatment response prediction [47] Survival prediction [48] Survival prediction [49]

Disease classification [50] Disease classification [51] Lesion classification [52] Lesion classification [53]

Disease classification [54] Cell classification [55]

CADx, computer-aided diagnosis; CADe, computer-aided detection; MRI, magnetic resonance imaging; CT, computed tomography; PET, positron emission tomography; OCT, optical coherence tomography; RGB, red green blue.

6


Jonghoon Kim, et al. predict a cognitive and motor developmental outcome score

or organ regeneration [55].

ing. Yasaka et al. [45] assessed the performance for predict-

Image processing applications using CNN architecture

etic acid-enhanced hepatobiliary phase MRI. In the above

Table 2.

by using structural brain networks of diffusion tensor imag-

ing the staging of liver fibrosis using a CNN model on gadoxtwo published papers, the predicted scores derived from

Image processing applications based on CNN are shown in

Image segmentation : The segmentation task partitions the

CNN correlated well with the ground-truth. Some studies ad-

imaging data into the target and background region. In med-

more, CNN-based approaches have shown potential in the

are related to a disease condition (i.e., finding the tumor re-

opted CNN to predict treatment response [46,47]. Furtherprediction of a patient’s survival and thus could be promising tools for precision medicine [48,49].

Classification : The classification task determines which

classes the imaging data belong to. The classes could be bi-

nary (e.g., normal and diseased) or multiple categories (e.g.,

subclasses within the given diseased condition) depending on the task. Many studies successfully applied CNN to classify the severity of disease for different organs (i.e., brain, lung,

and liver) using CT imaging [50-52]. Mohamed et al. [53] ap-

plied CNN to mammogram imaging and categorized data

ical imaging, this typically amounts to finding regions that gion). The task assigns voxels to binary classes using intensity, texture, and other derived information. Many studies successfully used CNN to segment the target region using CT

and MRI for various organs, such as the brain, liver, kidney, and prostate [56-61]. Fang et al. [62] proposed an automatic

segmentation algorithm of retina layer boundaries using

CNN for high-resolution optical coherence tomography imaging. Xu et al. [63] adopted CNN architecture for segmenting epithelial and stromal regions in histology images.

Image registration : The image registration task spatially

into a few classes proportional to breast density. Esteva et al.

aligns one image with another so that both can be compared

and even suggested that such operation could be performed

form. Many studies adopt multimodal imaging and thus reg-

[54] classified skin cancer photographs into binary classes using portable devices such as smartphones. In histology,

one study showed that cell differentiation could be classified

with CNN, and the results have the potential to identify tissue

to a common spatial framework through a geometric transistration is necessary to extract spatially aligned features in

those studies. Registration requires a long computational time as it involves a high degree of freedom (DOF) optimiza-

Table 2. Image processing applications using convolutional neural network architecture Task

Segmentation

Modalities

Brain

Tissue segmentation [56]

CT

Liver

Organ segmentation [57]

MRI

MRI CT

OCT

Enhancement, generation, and reconstruction

Histology image X-ray MRI MRI

X-ray MRI MRI

Diffusion MRI MRI (3T)

Prostate Bladder Heart

Kidney Retina Cell

Knee

Abdomen Brain Brain CT CT

Diffusion MRI MRI (7T)

MRI, magnetic resonance imaging; CT, computed tomography; OCT, optical coherence tomography.

http://doi.org/10.23838/pfm.2018.00030

Clinical goal

MRI

CT

Registration

Object

Organ segmentation [59] Organ segmentation [58]

Lesion segmentation [60] Organ segmentation [61]

Tissue segmentation [62] Tissue segmentation [63] Organ registration [64]

Region registration [65] Organ registration [66]

Image enhancement [67] Image generation [68] Image generation [69]

Image enhancement [70]

Image reconstruction [71]

7

Deep learning for medical imaging tion problem. The CNN-based approaches could be used as

is exploited using CNN, and temporal information is often ex-

landmarks or control points through CNN architecture. Miao

combination of CNN and RNN so that joint modeling of spatial

an alternative for the difficult registration problem by finding

et al. [64] improved the registration in terms of the computa-

tion time and capture range using a real-time 2D/3D registra-

tion framework based on CNN. Another study proposed a technique for correcting respiratory motion during free-breathing MRI using CNN [65]. Yang et al. [66] introduced

ploited using RNN. Studies using dynamic imaging adopted a and temporal information was possible references [72-77].

CAD application using RNN architecture

CADx/CADe applications based on RNN are shown in Table 3.

Detection : Xu et al. [72] proposed an end-to-end DL frame-

a novel registration framework based on CNN, Quicksilver,

work to accurately detect myocardial infarction at the pixel

Image enhancement : The image enhancement task aims to

based on CNN and the detected ROI was further analyzed us-

which performed fast predictive image registration.

improve the image quality of the objects of interest. Depend-

ing on the application, enhancement might occur in the context of image generation or reconstruction. Nonetheless, the

goal is to improve the image quality. Two studies proposed algorithms to generate a high-quality image (e.g., high dose CT) from a low quality image (e.g., low dose CT) using CNN,

which reduced noise and provided more structural informa-

level. In this framework, the location of the heart was found ing the dynamic cardiac MRI incorporating temporal motion

information using RNN. In the same manner, Chen et al. [73] used the combined CNN and RNN for a fetal ultrasound,

where ROI was found using CNN, and temporal information

was processed by RNN based on the features of the ROIs in consecutive frames.

Prediction : Han and Kamdar [74] proposed convolutional

tion than the low-quality images [67]. Image generation

recurrent neural networks (CRNN) for predicting methylation

the improved image quality by combining the distinctive in-

convolution layer extracted sequential feature vectors from

based on the synthesis of different modalities contributed to formation in each modality [68,69]. Golkov et al. [70] pro-

posed a reconstruction model based on CNN for shorter scanning time in diffusion MRI. This model allowed accelera-

status in glioblastoma patients. In the CRNN architecture, the

each MRI slice and RNN was used to predict the methylation status integrating sequential feature information.

Classification : Gao et al. [75] designed a combined architec-

tion of the scan time and obtaining quantitative diffusion

ture of CNN and RNN for classifying the severity of nuclear

(T)-like MRI from routine 3T MRI based on CNN, where recon-

were extracted from CNN, and then the features were hierar-

MRI. Bahrami et al. [71] explored the generation of 7 Tesla structed images benefited from both resolution and contrast compared to the 3T MRI.

RNN integrates results from the previous states and current

input and thus is suitable for tasks involving sequential or dynamic data. In medical image analysis, spatial information Table 3. CADx/CADe applications using recurrent neural network architecture Detection Prediction

Classification

Modalities

Object

Clinical goal

MRI

Heart


MRI

Brain

Status prediction [74]

US

OCT MRI

Infancy Eye

Heart

Object detection [73]

Disease classification [75] Lesion classification [76]

CADx, computer-aided diagnosis; CADe, computer-aided detection; MRI, magnetic resonance imaging; US, ultrasound; OCT, optical coherence tomography.

8

chically merged into high-order image-level features. Xue et al. [76] proposed a deep multitask relationship learning network for quantifying the left ventricle in cardiac imaging,

Applications using RNN architecture

Task

cataracts in slit-lamp images, where hierarchical feature sets

where cardiac sequence features were extracted using CNN and then temporally learned by RNN.

Image processing applications using RNN architecture

Imaging processing applications based on RNN are shown in Table 4.

Table 4. Image processing applications using recurrent neural network architecture Task

Segmentation Enhancement, generation, and reconstruction

Modalities

Object

Clinical goal

MRI

Brain

Lesion segmentation [77]

MRI

MRI

Image reconstruction [79]

US

Prostate Organ segmentation [78]

MRI, magnetic resonance imaging; US, ultrasound.


Jonghoon Kim, et al.

Image segmentation : Zhao et al. [77] proposed a tumor

Classification : Chen et al. [82] proposed an unsupervised

segmentation method based on DL using multimodal brain

learning framework based on the convolutional AE neural

CNN and RNN architectures in which CNNs assigned the seg-

for feature representation and classification for lung nodules.

MRI. They used a network composed of the combination of

mentation label to each pixel, and then RNNs optimized the segmentation result using both assigned labels and input

images. Yang et al. [78] proposed an automatic prostate segmentation method using only the RNN architecture. The

network in which the patch images from raw CT were used Cheng et al. [83] applied a stacked denoising AE to two differ-

ent modalities, ultrasound and CT. The aim was to classify benign and malignant lesions from the two modalities.

static ultrasound image was transformed into an interpreta-

Image processing applications using SAE architecture

used as input to the RNN.

Table 6.

ble sequence and then the extracted sequential data was

Image enhancement : Qin et al. [79] suggested a framework

Image processing applications based on SAE are shown in

Image segmentation : Zhao et al. [84] applied SAE to learn

to reconstruct high-quality cardiac MRI images from un-

compact feature representation from cryosection brain im-

reconstruction accuracy and speed by considering both spa-

ter, and cerebrospinal fluid.

der-sampled k-space data. They adopted CRNN to improve tial and temporal information.

SAE encodes the input data using a small number of features and then decodes them back to the dimension of the original input data. In medical imaging, SAEs are commonly used to

identify a compact set of features to represent the input data. The learned features could be used for specific image enhancement and reconstruction tasks.

CADx/CADe applications based on SAE are shown in Table 5.

Detection : One study showed that a stacked sparse AE could

be used for nuclei detection in breast cancer histopathology

images [80]. This architecture could capture high-level fea-

ture representations of pixel intensity and thus enabled the

classification task to be performed effectively for differentiating multiple nuclei.

Prediction : He et al. [81] adopted a variant of SAE to predict

cognitive deficits in preterm infants using functional MRI data.

Table 5. CADx/CADe applications using stacked auto-encoder architecture Detection

Prediction

Histology image MRI

Classification CT

US, CT

Object

Clinical goal

Breast


Lung

Lesion classification [82]

Brain

Breast

Risk prediction [81]

Lesion classification [83]

CADx, computer-aided diagnosis; CADe, computer-aided detection; MRI, magnetic resonance imaging; CT, computed tomography; US, ultrasound.

http://doi.org/10.23838/pfm.2018.00030

es. Benou et al. [86] adopted SAE to model the spatiotemporal structure in dynamic MRI, which led to denoised MRI. The

denoised MRI resulted in a more robust estimation of phar-

macokinetic parameters for blood-brain barrier quantification. Some adopted SAE to generate pseudo-CT scans from

MRI, which were used to reconstruct PET imaging with better tissue characteristics [87].

Among the discussed three network architectures, CNN is

CAD application using SAE architecture

Modalities

Image enhancement : Janowczyk et al. [85] applied sparse

AE to reduce the effects of color variation in histology imag-

Applications using SAE architecture

Task

aging and then classified voxels into white matter, gray mat-

the most widely used. Many studies adopted CNN for various target organs (brain, lung, breast, etc.) using various imaging modalities (MRI, CT, etc.). This is partly due to the fact that CNN processes information in a hierarchical manner similar

to visual processing in the human brain. RNN is typically used in conjunction with CNN, where the learned CNN fea-

tures were sequentially handled through the RNN. SAE has

strengths in compact feature representation, which is useful

in denoising and reconstruction tasks. In summary, there is no one network structure that solves all the medical imaging

Table 6. Image processing application using stacked auto-encoder architecture Task

Modalities Object

Enhancement, generation, and reconstruction

Histology image

Segmentation

Clinical goal

RGB

Brain

Tissue segmentation [84]

MRI

Brain


MRI

Tissue

CT


Image generation [87]

RGB, red green blue; MRI, magnetic resonance imaging; CT, computed tomography.

9

Deep learning for medical imaging problems, and thus, researchers should choose the appropriate network architecture suitable for a given problem.

SOFTWARE TOOLS FOR DEEP LEARNING

Researchers need dedicated software tools to perform DL research. Fortunately, much of the DL software is open-source; thus, the studies are more reproducible and cost effective.

We review commonly used open-source DL software tools in this section.

Caffe is one of the early DL tools developed by Berkeley Vi-

sion and Learning Center [87]. The tool emphasized modularity and speed based on C++ and Python. Many early CNN

works have been performed with Caffe. Researchers could

ing research, the large sample requirement could be partly

alleviated by multisite data acquisition. Researchers need to apply a standardized protocol to acquire data so that they

can be effectively combined into one set of training data for

DL. Another way to boost training samples is to use an open

research database. The Human Connectome Project houses thousands of high-quality brain MRI that are open to the research community. A potential issue with mixing data from

different sites or research databases is the heterogeneity in image quality. One can render high-quality data into low-

quality data so that all data are of similar quality. This is the

practical approach because rendering low-quality data into high-quality data is difficult.

Many DL methods belong to the supervised approach;

easily transfer the learned models (i.e., layer structure and

hence, they require manual labeling. Labeling thousands of

ize their own DL model. Such procedures are often referred

that, inter- and intraobserver variability need to be consid-

weights) from other research projects and use them to initialto as transfer learning, and they typically save computation

time in training the model. Theano is another early DL tool [88]. Theano is Python-based and is efficient in computation using a GPU and central processing unit together.

Tensorflow is one of the most widely used tools for DL and

it is backed by the internet search giant Google [89]. The Google Brain Team developed the software and it is based on Python. Users are free to leverage the vast software library

available in Python. The software is well-maintained and updated frequently for additional functionality.

Torch is a DL tool that aims to facilitate the procedures to

imaging data by human experts is cumbersome. On top of ered, which makes the labeling procedure even more problematic. One possible way to tackle this issue is to apply a

two-tiered semiautomatic approach [43]. An automatic algorithm would perform the first round of labeling and then the

human experts can either accept or modify the results of the first round. Recently, one study proposed a DL method to au-

tomatically retrieve images from a large database that matched human set criterion [92-94]. This approach has the potential to be used as the initial labeling for further refinement.

Another issue related to sample size is the lack of balance in

build complex models. It was originally based on non-Python

comparison groups. Many studies compare two groups: the

proved user friendliness [90].

sites have many samples of the disease group, while

Lua, but recently it added Python support via PyTorch for im-

LIMITATIONS AND FUTURE DIRECTIONS

Modern DL methods learn very high DOF models (i.e., millions of weights) from the training data. Hence, the sheer

number of required training data is very high compared to

conventional machine learning methods. Recent DL applications in brain MRI learned models from more than 1.2 million training data [91]. There are algorithm enhancements, such

as augmentation, to artificially boost the number of training

samples, but the quality of the DL methods directly rely on the number and quality of training samples. This is one of

the biggest hurdles of DL research in medical imaging and is

one of the reasons why large corporations such as Google can produce high-quality DL models, as they already have a

huge number of training samples in-house. In medical imag10

normal control group and the diseased group. Many clinical

matched controls could be lacking. Ideally, DL algorithms require an equal number of samples in the comparison groups.

If there is a large imbalance between the comparison groups,

the DL algorithm would not be able to fully learn the un-

der-represented group. A good practice is to prospectively plan research projects to avoid imbalance in comparison

groups as much as possible. Many diseased conditions we explored are in fact quite heterogeneous. A given disease

condition might consists of many subtypes, and more studies are aiming to dissect the differences among the subtypes.

Researchers should make sure that they have appropriate la-

bels for the subtypes and enough samples of each subtype so that DL models could be applied effectively.

DL is a highly flexible modeling approach to learn an inher-

ent representation of the input data. It optimizes a loss function to find millions of weights that can best explain the in-


Jonghoon Kim, et al. put training data. It is very difficult to interpret how certain

DL approaches, as DL has shown remarkable performance in

left with a black box that performs the given task really well

tional machine learning approaches. In this review paper, we

weights contribute to the final model. In many cases, we are [95,96]. In traditional machine learning, we could quantify how each semantic feature contributes to the final model and better understand or improve the model as necessary. For example, we know that an irregular tumor boundary is

strongly linked with malignant tumors [97,98]. However, for DL, we cannot say the n-th weight in the j-th layer has a strong link with the final outcome. Such interpretability is

largely lacking in DL. Ribeiro et al. [99] proposed a novel explanation technique, local interpretable model-agnostic explanations, which performs a local approximation of the

complex DL model. This algorithm carries out linear classifi-

cation based on distinctive features from the simplified model, which might be interpretable. Some studies surveyed vi-

sual interpretability of various DL architectures and showed that the DL model is reliable across many domains [100-102]. There is active research tackling this interpretability issue

and researchers should pay attention to future developments.

A DL algorithm requires many tuning parameters to proper-

ly train the model. The hyperparameters include learning rate, dropout rate, and kernel functions. A slight modification

of these parameters might lead to drastically different mod-

els with varying performances. So far, these parameters have

been chosen based on heuristic approaches relying on the experience of human experts. Active research is being performed regarding methods to automatically find optimal hyperparameters for DL. Domhan et al. [103] developed a

method for automatically optimizing the hyperparameters, where the process was based on the inference of perfor-

mance using a learning curve from the first few iterations. They would terminate the model, leading to bad performance based on the few initial data points. Shen et al. [104]

proposed an intelligent system, including a parameter-tuning policy network, which determines the direction and magnitude for fine-tuning hyperparameters.

CONCLUSION

non-medical regular imaging research compared to conventouched on a brief history from conventional machine learn-

ing to DL, surveyed many DL applications in medical imaging, and concluded with limitations and future directions of DL in medical imaging. Despite the limitations, the advantag-

es of DL far outweigh its shortcomings, and thus, it will be an

essential tool for diagnosis and prognosis in the era of preci-

sion medicine. Future research teams in medical imaging

should integrate DL experts in addition to clinical scientists in their teams so as to fully harness the potential of DL.

CONFLICTS OF INTEREST

No potential conflict of interest relevant to this article was reported.

ACKNOWLEDGMENTS

This work was supported by the Institute for Basic Science

(grant number IBS-R015-D1) and the National Research Foundation of Korea (NRF) (grant number NRF-2016R1A2B4008545).

REFERENCES

1. Wang S, Summers RM. Machine learning and radiology. Med Image Anal 2012;16:933-51.

2. de Bruijne M. Machine learning approaches in medical

image analysis: from detection to diagnosis. Med Image Anal 2016;33:94-7.

3. Wernick MN, Yang Y, Brankov JG, Yourganov G, Strother

SC. Machine learning in medical imaging. IEEE Signal Process Mag 2010;27:25-38.

4. Gillies RJ, Kinahan PE, Hricak H. Radiomics: images are more than pictures, they are data. Radiology 2016;278: 563-77.

5. Aerts HJ, Velazquez ER, Leijenaar RT, Parmar C, Gross-

mann P, Carvalho S, et al. Decoding tumour phenotype

by noninvasive imaging using a quantitative radiomics approach. Nat Commun 2014;5:4006.

DL is already widespread, and it will continue to grow in the

6. Lambin P, Leijenaar RTH, Deist TM, Peerlings J, de Jong

ing new interactions among different fields. Experts in ma-

tween medical imaging and personalized medicine. Nat

near future in all fields of science. The advent of DL is foster-

chine learning (mostly from computer science) are actively embedded in research teams to solve critical medical problems. Medical image processing will benefit immensely from http://doi.org/10.23838/pfm.2018.00030

EEC, van Timmeren J, et al. Radiomics: the bridge beRev Clin Oncol 2017;14:749-62.

7. van Ginneken B, Schaefer-Prokop CM, Prokop M. Computer-aided diagnosis: how to move from the laborato11

Deep learning for medical imaging ry to the clinic. Radiology 2011;261:719-32.

8. Doi K. Computer-aided diagnosis in medical imaging:

historical review, current status and future potential. Comput Med Imaging Graph 2007;31:198-211.

9. Park SH, Han K. Methodologic guide for evaluating clini-

cal performance and effect of artificial intelligence tech-

20. Rosenblatt F. Report no. 85-460-1. The perceptron, a

perceiving and recognizing automaton. Buffalo (NY): Cornell Aeronautical Laboratory; 1957.

21. Rumelhart DE, Hinton GE, Williams RJ. Learning representations by back-propagating errors. Nature 1986; 323:533-6.

nology for medical diagnosis and prediction. Radiology

22. Le QV, Ngiam J, Coates A, Lahiri A, Prochnow B, Ng AY.

10. McCulloch WS, Pitts W. A logical calculus of the ideas

L, Scheffer T, editors. Proceedings of the 28th Interna-

2018;286:800-9.

immanent in nervous activity. Bull Math Biophys 1943;5: 115-33.

11. Van Der Malsburg C. Frank Rosenblatt: principles of neu-

tional Conference on Machine Learning; 2011 Jun 28-Jul

2; Bellevue, WA. Madison (WI): Omnipress; 2011. p. 26572.

rodynamics: perceptrons and the theory of brain mech-

23. Schmidhuber J. Deep learning in neural networks: an

1984 Oct 1-4; Trieste (IT): Springer Berlin Heidelberg;

24. Rumelhart DE, McClelland JL; University of California,

anisms. In: Palm G, Aertsen A, editors. Brain theory; 1986. p. 245-8.

12. Rumelhart DE, Hinton GE, Williams RJ. Learning internal

overview. Neural Netw 2015;61:85-117.

San Diego, PDP Research Group. Parallel distributed processing. Cambridge (MA): MIT press; 1986. p. 567.

representations by error propagation. In: Anderson JA,

25. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;

research. Cambridge (MA): MIT Press; 1988. p. 696-9.

26. Bengio Y, Courville A, Vincent P. Representation learn-

Rosenfeld E, editors. Neurocomputing: foundations of 13. Nasrabadi NM. Pattern recognition and machine learning. J Electron Imaging 2007;16:49901.

14. Kotsiantis SB. Supervised machine learning: a review of classification techniques. Informatica 2007;31:249-68.

15. Kuang D, Guo X, An X, Zhao Y, He L. Discrimination of

521:436-44.

ing: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell 2013;35:1798-828.

27. Glorot X, Bengio Y. Understanding the difficulty of train-

ing deep feedforward neural networks. Proc Mach Learn Res 2010;9:249-56.

ADHD based on fMRI data with deep belief network. In:

28. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classifi-

ing in bioinformatics; 2014 Aug 3-6; Taiyuan, CN. Cham

Pereira F, Burges CJC, Bottou L, Weinberger KQ, editors.

Huang DS, Han K, Gromiha M, editors. Intelligent comput(CH): Springer International Publishing; 2014. p. 225-32.

16. Kuang D, He L. Classification on ADHD with deep learning. Proceedings of the 2014 International Conference

on Cloud Computing and Big Data (CCBD); 2014 Nov 12-

14; Wuhan, CN. Washington, DC: IEEE Computer Society; 2014. p. 27-32.

17. Lam C, Yu C, Huang L, Rubin D. Retinal lesion detection with deep learning using image patches. Invest Ophthalmol Vis Sci 2018;59:590-6.

18. Liu M, Zhang J, Adeli E, Shen D. Deep multi-task multi-

channel learning for joint classification and regression of brain status. In: Descoteaux M, Maier-Hein L, Franz A,

cation with deep convolutional neural networks. In: Advances in Neural Information Processing Systems 25 (NIPS 2012); 2012 Dec 3-6; Lake Tahoe, NV. Red Hook (NY): Curran Associates Inc.; 2013. p. 1097-105.

29. LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based

learning applied to document recognition. Proc IEEE Inst Electr Electron Eng 1998;86:2278-324.

30. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, et al. Going deeper with convolutions. In: IEEE Computer Society, editor. 2015 IEEE Conference on Computer

Vision and Pattern Recognition (CVPR); 2015 Jun 7-12; Boston, MA. Los Alamitos (CA): IEEE Computer Society; 2015. p. 1-9.

Jannin P, Collins DL, Duchesne S, editors. Medical Image

31. Simonyan K, Zisserman A. Very deep convolutional net-

CAI 2017; 2017 Sep 11-13; Quebec City, QC. Cham (CH):

Conference on Learning Representations 2015; 2015

Computing and Computer-Assisted Intervention: MIC-

Springer International Publishing; 2017. p. 3-11.

19. Hagan MT, Demuth HB, Beale MH, De Jesus O. Neural

network design. 2nd ed. Boston (MA): PWS Publishing; 2014.

12

On optimization methods for deep learning. In: Getoor

works for large-scale image recognition. International

May 7-9; San Diego, CA. Computational and Biological Learning Society; 2015.

32. He K, Zhang X, Ren S, Sun J. Deep residual learning for

image recognition. Proc IEEE Comput Soc Conf Comput http://pfmjournal.org

Jonghoon Kim, et al. Vis Pattern Recognit 2016;2016:770-8.

a convolutional neural network for detecting gastric can-

33. Hinton GE, Zemel RS. Autoencoders, minimum descrip-

cer in endoscopic images. Gastric Cancer. 2018 Jan 15

sauro G, Alspector J, editors. Advances in Neural Infor-

43. Albarqouni S, Baur C, Achilles F, Belagiannis V, Demirci S,

Dec 2; Denver, CO. San Francisco (CA): Morgan Kaufmann

detection in breast cancer histology images. IEEE Trans

tion length and helmholtz free energy. In: Cowan JD, Te-

mation Processing Systems 6 (NIPS 1993); 1993 Nov 29Publishers Inc.; 1994. p. 3-10.

34. Bourlard H, Kamp Y. Auto-association by multilayer perceptrons and singular value decomposition. Biol Cybern 1988;59:291-4.

35. Bengio Y, Lamblin P, Popovici D, Larochelle H. Greedy

[Epub]. https://doi.org/10.1007/s10120-018-0793-2.

Navab N. AggNet: deep learning from crowds for mitosis Med Imaging 2016;35:1313-21.

44. Kawahara J, Brown CJ, Miller SP, Booth BG, Chau V,

Grunau RE, et al. BrainNetCNN: convolutional neural networks for brain networks: towards predicting neurodevelopment. Neuroimage 2017;146:1038-49.

layer-wise training of deep networks. In: Scholkopf B,

45. Yasaka K, Akai H, Kunimatsu A, Abe O, Kiryu S. Liver fi-

mation Processing Systems 19 (NIPS 2006); 2006 Dec

by using gadoxetic acid-enhanced hepatobiliary phase

Platt JC, Hofmann T, editors. Advances in Neural Infor-

4-7; Vancouver, BC. Cambridge (MA): MIT Press; 2006. p. 153-60.

36. Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol PA. Stacked denoising autoencoders: learning useful repre-

sentations in a deep network with a local denoising criterion. J Mach Learn Res 2010;11:3371-408.

37. Graves A, Jaitly N. Towards end-to-end speech recognition with recurrent neural networks. In: International Machine Learning Society, editor. Proceedings of the 31st In-

ternational Conference on International Conference on

Machine Learning. Volume 32; 2014 Jun 21-26; Beijing, CN. Stroudsburg (PA): International Machine Learning Society; 2014. p. II-1764-72.

38. Graves A. Supervised sequence labelling. In: Graves A, editor. Supervised sequence labelling with recurrent

neural networks. Berlin (DE): Springer Berlin Heidelberg; 2012. p. 5-13.

39. Dou Q, Chen H, Yu L, Zhao L, Qin J, Wang D, et al. Automatic detection of cerebral microbleeds from MR images

via 3D convolutional neural networks. IEEE Trans Med Imaging 2016;35:1182-95.

brosis: deep convolutional neural network for staging MR images. Radiology 2018;287:146-55.

46. Ypsilantis PP, Siddique M, Sohn HM, Davies A, Cook G, Goh V, et al. Predicting response to neoadjuvant chemo-

therapy with PET imaging using convolutional neural networks. PLoS One 2015;10:e0137036.

47. Prahs P, Radeck V, Mayer C, Cvetkov Y, Cvetkova N, Hel-

big H, et al. OCT-based deep learning algorithm for the

evaluation of treatment indication with anti-vascular endothelial growth factor medications. Graefes Arch Clin Exp Ophthalmol 2018;256:91-8.

48. van der Burgh HK, Schmidt R, Westeneng HJ, de Reus MA, van den Berg LH, van den Heuvel MP. Deep learning predictions of survival based on MRI in amyotrophic lateral sclerosis. Neuroimage Clin 2016;13:361-9.

49. Oakden-Rayner L, Carneiro G, Bessen T, Nascimento JC,

Bradley AP, Palmer LJ. Precision radiology: predicting longevity using feature engineering and deep learning

methods in a radiomics framework. Sci Rep 2017;7:1648.

50. Gao XW, Hui R, Tian Z. Classification of CT brain images based on deep learning networks. Comput Methods Programs Biomed 2017;138:49-56.

40. Zreik M, Lessmann N, van Hamersvelt RW, Wolterink JM,

51. Anthimopoulos M, Christodoulidis S, Ebner L, Christe A,

the myocardium in coronary CT angiography for identi-

tial lung diseases using a deep convolutional neural

Voskuil M, Viergever MA, et al. Deep learning analysis of fication of patients with functionally significant coro-

nary artery stenosis. Med Image Anal 2018;44:72-85.

41. Xu L, Tetteh G, Lipkova J, Zhao Y, Li H, Christ P, et al. Automated whole-body bone lesion detection for multiple myeloma on (68)Ga-pentixafor PET/CT imaging using deep

learning methods. Contrast Media Mol Imaging 2018; 2018:2391925. https://doi.org/10.1155/2018/2391925.

42. Hirasawa T, Aoyama K, Tanimoto T, Ishihara S, Shichijo S, Ozawa T, et al. Application of artificial intelligence using

http://doi.org/10.23838/pfm.2018.00030

Mougiakakou S. Lung pattern classification for interstinetwork. IEEE Trans Med Imaging 2016;35:1207-16.

52. Yasaka K, Akai H, Abe O, Kiryu S. Deep learning with

convolutional neural network for differentiation of liver masses at dynamic contrast-enhanced CT: a preliminary study. Radiology 2018;286:887-96.

53. Mohamed AA, Berg WA, Peng H, Luo Y, Jankowitz RC, Wu S. A deep learning method for classifying mammographic breast density categories. Med Phys 2018;45:314-21.

54. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau 13

Deep learning for medical imaging HM, et al. Dermatologist-level classification of skin can-

based image registration: a feasibility study. Br J Radiol

55. Niioka H, Asatani S, Yoshimura A, Ohigashi H, Tagawa S,

66. Yang X, Kwitt R, Styner M, Niethammer M. Quicksilver:

cer with deep neural networks. Nature 2017;542:115-8.

Miyake J. Classification of C2C12 cells at differentiation by convolutional neural network of deep learning using phase contrast images. Hum Cell 2018;31:87-93.

56. Zhang W, Li R, Deng H, Wang L, Lin W, Ji S, et al. Deep convolutional neural networks for multi-modality isointense infant brain image segmentation. Neuroimage 2015;108:214-24.

57. Hu P, Wu F, Peng J, Liang P, Kong D. Automatic 3D liver

segmentation based on deep learning and globally op-

timized surface evolution. Phys Med Biol 2016;61:867698.

58. Cha KH, Hadjiiski L, Samala RK, Chan HP, Caoili EM, Co-

han RH. Urinary bladder segmentation in CT urography using deep-learning convolutional neural network and level sets. Med Phys 2016;43:1882.

59. Tian Z, Liu L, Zhang Z, Fei B. PSNet: prostate segmentation on MRI based on a convolutional neural network. J

Med Imaging (Bellingham) 2018;5:021208. https://doi. org/10.1117/1.jmi.5.2.021208.

60. Tan LK, Liew YM, Lim E, McLaughlin RA. Cardiac left ven-

tricle segmentation using convolutional neural network regression. In: IEEE Engineering in Medicine and Biology

Society, editor. 2016 IEEE EMBS Conference on Biomedical Engineering and Sciences (IECBES); 2016 Dec 4-8;

Kuala Lumpur, ML. Piscataway (NJ): IEEE; 2016. p. 490-3.

61. Sharma K, Rupprecht C, Caroli A, Aparicio MC, Remuzzi

fast predictive image registration: a deep learning approach. Neuroimage 2017;158:378-96.

67. Yang X, De Andrade V, Scullin W, Dyer EL, Kasthuri N, De

Carlo F, et al. Low-dose X-ray tomography through a deep convolutional neural network. Sci Rep 2018;8:2575.

68. Han X. MR-based synthetic CT generation using a deep convolutional neural network method. Med Phys 2017; 44:1408-19.

69. Roy S, Butman JA, Pham DL. Synthesizing CT from ul-

trashort echo-time MR images via convolutional neural networks. In: Tsaftaris SA, Gooya A, Frangi AF, Prince JL,

editors. Simulation and synthesis in medical imaging. Cham (CH): Springer International Publishing; 2017. p. 24-32.

70. Golkov V, Dosovitskiy A, Sperl JI, Menzel MI, Czisch M,

Samann P, et al. Q-space deep learning: twelve-fold shorter and model-free diffusion MRI scans. IEEE Trans Med Imaging 2016;35:1344-51.

71. Bahrami K, Rekik I, Shi F, Shen D. Joint reconstruction

and segmentation of 7T-like MR images from 3T MRI based on cascaded convolutional neural networks. In:

Descoteaux M, Maier-Hein L, Franz A, Jannin P, Collins DL, Duchesne S, editors. Medical Image Computing and

Computer-Assisted Intervention: MICCAI 2017; 2017 Sep

11-13; Quebec City, QC. Cham (CH): Springer International Publishing; 2017. p. 764-72.

A, Baust M, et al. Automatic segmentation of kidneys us-

72. Xu C, Xu L, Gao Z, Zhao S, Zhang H, Zhang Y, et al. Direct

in autosomal dominant polycystic kidney disease. Sci

deep-learning algorithm. In: Descoteaux M, Maier-Hein

ing deep learning for total kidney volume quantification Rep 2017;7:2049.

62. Fang L, Cunefare D, Wang C, Guymer RH, Li S, Farsiu S.

Automatic segmentation of nine retinal layer boundar-

ies in OCT images of non-exudative AMD patients using

deep learning and graph search. Biomed Opt Express 2017;8:2732-44.

63. Xu J, Luo X, Wang G, Gilmore H, Madabhushi A. A deep convolutional neural network for segmenting and clas-

sifying epithelial and stromal regions in histopathological images. Neurocomputing 2016;191:214-23.

64. Miao S, Wang ZJ, Liao R. A CNN regression approach for

real-time 2D/3D registration. IEEE Trans Med Imaging 2016;35:1352-63.

65. Lv J, Yang M, Zhang J, Wang X. Respiratory motion cor-

rection for free-breathing 3D abdominal MRI using CNN-

14

2018;91:20170788. https://doi.org/10.1259/bjr.20170788.

detection of pixel-level myocardial infarction areas via a L, Franz A, Jannin P, Collins DL, Duchesne S, editors.

Medical Image Computing and Computer-Assisted In-

tervention: MICCAI 2017; 2017 Sep 11-13; Quebec City, QC. Cham (CH): Springer International Publishing; 2017. p. 240-9.

73. Chen H, Dou Q, Ni D, Cheng JZ, Qin J, Li S, et al. Auto-

matic fetal ultrasound standard plane detection using

knowledge transferred recurrent neural networks. In: Navab N, Hornegger J, Wells WM, Frangi A, editors. Med-

ical Image Computing and Computer-Assisted Intervention: MICCAI 2015; 2015 Oct 5-9; Munich, DE. Cham (CH): Springer International Publishing; 2015. p. 507-14.

74. Han L, Kamdar MR. MRI to MGMT: predicting methylation

status in glioblastoma patients using convolutional recur-

rent neural networks. Pac Symp Biocomput 2018;23:331http://pfmjournal.org

Jonghoon Kim, et al. 42.

75. Gao X, Lin S, Wong TY. Automatic feature learning to grade nuclear cataracts based on deep learning. IEEE Trans Biomed Eng 2015;62:2693-701.

76. Xue W, Brahm G, Pandey S, Leung S, Li S. Full left ventri-

denoising of contrast-enhanced MRI sequences. Med Image Anal 2017;42:145-59.

87. Liu F, Jang H, Kijowski R, Bradshaw T, McMillan AB. Deep learning MR imaging-based attenuation correction for PET/MR imaging. Radiology 2018;286:676-84.

cle quantification via deep multitask relationships

88. The Theano Development Team, Al-Rfou R, Alain G,

77. Zhao X, Wu Y, Song G, Li Z, Zhang Y, Fan Y. A deep learn-

a Python framework for fast computation of mathemat-

learning. Med Image Anal 2018;43:54-65.

ing model integrating FCNNs and CRFs for brain tumor segmentation. Med Image Anal 2018;43:98-111.

78. Yang X, Yu L, Wu L, Wang Y, Ni D, Qin J, et al. Fine-grained

Almahairi A, Angermuller C, Bahdanau D, et al. Theano:

ical expressions [Internet]. Ithaca (NY): Cornell Universi-

ty; 2016 [cited 2018 Apr 23]. Available from: https://arxiv. org/abs/1605.02688.

recurrent neural networks for automatic prostate seg-

89. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, et

S, editors. Thirty-First AAAI Conference on Artificial In-

ing. In: Keeton K, Roscoe T, editors. OSDI’16 Proceedings

mentation in ultrasound images. In: Singh S, Markovitch telligence (AAAI-17); 2017 Feb 4-9; San Francisco, CA. Palo Alto (CA): AAAI Press; 2017. p. 1633-9.

79. Qin C, Schlemper J, Caballero J, Price A, Hajnal JV, Ruec-

al. TensorFlow: a system for large-scale machine learn-

of the 12th USENIX conference on Operating Systems Design and Implementation; 2016 Nov 2-4; Savannah, GA. Berkeley (CA): USENIX Association; 2016. p. 265-83.

kert D. Convolutional recurrent neural networks for dy-

90. Collobert R, van der Maaten L, Joulin A. Torchnet: an

Cornell University; 2017 [cited 2018 Apr 23]. Available

Balcan MF, Weinberger KQ, editors. Proceedings of the

namic MR image reconstruction [Internet]. Ithaca (NY): from: https://arxiv.org/abs/1712.01751.

80. Xu J, Xiang L, Liu Q, Gilmore H, Wu J, Tang J, et al. Stacked

Sparse Autoencoder (SSAE) for nuclei detection on breast cancer histopathology images. IEEE Trans Med Imaging 2016;35:119-30.

81. He L, Li H, Holland SK, Yuan W, Altaye M, Parikh NA. Early prediction of cognitive deficits in very preterm infants

using functional connectome data in an artificial neural network framework. NeuroImage Clin 2018;18:290-7.

82. Chen M, Shi X, Zhang Y, Wu D, Guizani M. Deep features learning for medical image analysis with convolutional autoencoder neural network. IEEE Trans Big Data. 2017 Jun 20 [Epub]. https://doi.org/10.1109/TBDATA.2017.2717439.

open-source platform for (deep) learning research. In: 33rd International Conference on Machine Learning

(ICML-2016); 2016 Jun 19-24; New York, NY. Stroudburg

(PA): International Machine Learning Society; 2016. p. 19-24.

91. Wachinger C, Reuter M, Klein T. DeepNAT: deep convolu-

tional neural network for segmenting neuroanatomy. Neuroimage 2018;170:434-45.

92. Cao Y, Steffey S, He J, Xiao D, Tao C, Chen P, et al. Medi-

cal image retrieval: a multimodal approach. Cancer Inform 2015;13(Suppl 3):125-36.

93. Qayyum A, Anwar SM, Awais M, Majid M. Medical image

retrieval using deep convolutional neural network. Neurocomputing 2017;266:8-20.

83. Cheng JZ, Ni D, Chou YH, Qin J, Tiu CM, Chang YC, et al.

94. Sun Q, Yang Y, Sun J, Yang Z, Zhang J. Using deep learn-

ture: applications to breast lesions in US images and

TS, Zhang J, editors. Medical imaging 2017: imaging in-

Computer-aided diagnosis with deep learning architec-

pulmonary nodules in CT scans. Sci Rep 2016;6:24454.

84. Zhao G, Wang X, Niu Y, Tan L, Zhang SX. Segmenting brain

tissues from Chinese visible human dataset by deep-learned

ing for content-based medical image retrieval. In: Cook formatics for healthcare, research, and applications;

2017 Feb 11-16; Orlando, FL. Bellingham (WA): SPIE; 2017. p. 1013811-2. https://doi.org/10.1117/12.2251115.

features with stacked autoencoder. Biomed Res Int 2016;

95. Castelvecchi D. Can we open the black box of AI? Nature

85. Janowczyk A, Basavanhally A, Madabhushi A. Stain Nor-

96. Rodvold DM, McLeod DG, Brandt JM, Snow PB, Murphy

cation to digital pathology. Comput Med Imaging Graph

cians: taking the lid off the black box. Prostate 2001;46:39-

2016:5284586. http://dx.doi.org/10.1155/2016/5284586.

malization using Sparse AutoEncoders (StaNoSA): appli2017;57:50-61.

86. Benou A, Veksler R, Friedman A, Riklin Raviv T. Ensem-

ble of expert deep neural networks for spatio-temporal

http://doi.org/10.23838/pfm.2018.00030

2016;538:20-3.

GP. Introduction to artificial neural networks for physi44.

97. Li F, Sone S, Abe H, Macmahon H, Doi K. Malignant versus benign nodules at CT screening for lung cancer: compari15

Deep learning for medical imaging son of thin-section CT findings. Radiology 2004;233:7938.

98. You X, Sun X, Yang C, Fang Y. CT diagnosis and differentia-

a survey. Front Inf Technol Electron Eng 2018;19:27-39.

102. Liu X, Wang X, Matwin S. Interpretable deep convolutional neural networks via meta-learning [Internet].

tion of benign and malignant varieties of solitary fibrous tumor of the pleura. Medicine (Baltimore) 2017;96:e9058.

Ithaca (NY): Cornell University; 2018 [cited 2018 Apr 23]. Available from: https://arxiv.org/abs/1802.00560.

99. Ribeiro MT, Singh S, Guestrin C. “Why should I trust

103. Domhan T, Springenberg JT, Hutter F. Speeding up au-

Krishnapuram B, editor. Proceedings of the 22nd ACM

networks by extrapolation of learning curves. In: Yang Q,

you?”: explaining the predictions of any classifier. In:

tomatic hyperparameter optimization of deep neural

SIGKDD International Conference on Knowledge Dis-

Wooldridge MJ, editors. Proceedings of the 24th Inter-

covery and Data Mining; 2016 Aug 13-17; San Francisco,

national Conference on Artificial Intelligence; 2015 Jul

CA. New York (NY): Association for Computing Machinery; 2016. p. 1135-44.

25-31; Buenos Aires, AR. Palo Alto (CA): AAAI Press; 2015. p. 3460-8.

100. Hohman F, Kahng M, Pienta R, Chau DH. Visual analytics

104. Shen C, Gonzalez Y, Chen L, Jiang SB, Jia X. Intelligent

frontiers [Internet]. Ithaca (NY): Cornell University; 2018

construction via deep reinforcement learning [Internet].

in deep learning: an interrogative survey for the next

parameter tuning in optimization-based iterative CT re-

[cited 2018 Apr 23]. Available from: https://arxiv.org/

Ithaca (NY): Cornell University; 2017 [cited 2018 Apr 23].

abs/1801.06889.

Available from: https://arxiv.org/abs/1711.00414.

101. Zhang Q, Zhu S. Visual interpretability for deep learning:

16
