W. Clem Karl
M. Usman Ghani
Department of Electrical and Computer EngineeringDepartment of Biomedical EngineeringBoston University
Data and Image Domain Deep Learning for Tomographic Computational Imaging
Overview
Data and Image Domain Learning Tomography overview and some context
Some of Our Recent Work Data-domain Deep Learning
Metal artifact mitigation Sparse-view CT
Integrating Data Domain and Image Domain Learning Consensus Equilibrium Limited Angle CT application
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
2
Model: Goal make an image x from y Applications:
Security – a motivating example for us Medical Defense (e.g. SAR)
Computed Tomography
3
Physical SystemSinogram
DataImage
Rxy
x yR
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
Making Images with Learning
Image Domain vs Data Domain? Image Domain Learning: XX mappings Data Domain Learning: YY mappings
Most work uses Image Domain Learning Our focus today: Data Domain Learning
Good or bad idea? I hope to convince you it can be good!
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
4
R
Data DomainImage Domain
Image Reconstruction
Image Post-Processing
Data Pre-Processing
Raw Data Image
• Plug-and-Play• Proximal-like operators• Unrolled algorithms
• Image-domain DL• Data-domain DL
"" 1R
Case 1: Metal Artifact Reduction via Data-Domain Deep LearningM. U. Ghani and W. C. Karl, "Fast Enhanced CT Metal Artifact Reduction using Data Domain Deep Learning," in IEEE Transactions on Computational Imaging, 2019.
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
5
Applications with Metal Artifacts
Security
Medical Imaging
Esther Meyer, Rainer Raupach, Michael Lell, Bernhard Schmidt, and Marc Kachelrieß. Normalized metal artifact reduction (nmar) in computed tomography. Medical physics, 37(10):5482–5493, 2010.
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
Sinogram
Sinogram
6
CT Metal Artifact Reduction
Security Motivation Improve detection in X-ray-based CT scanners by reducing
metal artifacts Iterative reconstruction can reduce artifacts and improve
detection But… iterative reconstruction is slow, priors can be generic,
brittle
Idea: Use Data Domain Deep Learning Focus on pre-reconstruction correction of metal artifacts in
X-ray CT data Goal: A fast method within the standard workflow
“Bags per hour” matter Explore use of synthetic training data
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
7
Deep-MAR Framework Idea
Key points: Data-domain-focus – pose as problem of sinogram completion Learn complete sinogram behavior (vs patches) Use fully convolutional network (FCN) for efficiency Use C-GAN for good performance Synthetic data training, coupled with real data tuning
Speed vs Performance
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
8
Generator Network
Discriminator NetworkReal/Fake?
Conditional Generative Adversarial Network
Input Sinogram
FBP
Output Sinogram
Reconstruction without Metals
Deep-MAR Framework Details
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
9
Conditional Generative Adversarial Network (CGAN)
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
10
Generator Network
Discriminator NetworkReal/Fake?
Conditional Generative Adversarial Network
CGAN Data-Completion Network
Mean Squared Error (MSE) loss function over-smooths results in image-to-image tasks.
CGAN loss:
Generator loss:
y = incomplete raw data = complete data
Mask-specific data completion: loss focused on missing data
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
11
))](,(1[log(),(log(),( , yGyDEyyDDGL ycyycGAN c
])([),(maxminarg 2
2,* yGyDGLG cyycGANDG c
cy
Training Scanned bag data is limited
Perform majority of training on simulated data Fine tune with transfer learning on limited real data
Training only requires incomplete data pairs Material knowledge not required
Overall significant improvement vs training from scratch.
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
12
Example Dataset
Simulated Data Simulation setup based on Imatron C300 10,000 examples with up to 5 metallic objects 50,000 sinogram pairs
Transfer learning with real data 1,706 examples from ALERT TO4 data [1] at 130KeV 8,530 sinogram pairs
Images: 475 mm x 475 mm
Parallel beam re-binned data 1024 detectors 720 angular views
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
13[1] Crawford, C, “Advances in automatic target recognition (ATR) for CT based object detection system–Final report, Dept,” Homeland Security Center Excellence, Task order number HSHQDC–12–J–00429, Boston, MA, 2014.
Results – Simulated Example
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
14LI-MAR = Linear interpolation MARWNN-MAR = Weighted Nearest-Neighbor MARDeep-MAR = Deep Learning MAR
Input Target LI-MAR WNN-MAR Deep-MAR
Uncorrected LI-MAR WNN-MAR Deep-MAR
Results – Real Example
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
15LI-MAR = Linear interpolation MARWNN-MAR = Weighted Nearest-Neighbor MARDeep-MAR = Deep Learning MAR
Input Target LI-MAR WNN-MAR Deep-MAR
Uncorrected LI-MAR WNN-MAR Deep-MAR
Attention Maps
Occlude 11x11 patches with random U(0,max). Attention Maps: log-normalized resulting MSE Data near gaps most important Bigger gaps need more data Challenging problems use more of the sinogram – problem adaptive
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
16
Latent space analysis
Encoder representation t-SNE visualization [1]
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
17L. v. d. Maaten and G. Hinton, “Visualizing data using t-sne,” Journal of machine learning research, vol. 9, no. Nov, pp. 2579–2605, 2008.
Input projection data
Latent representation
Case 2: Low-Dose CT via Sparse-Projections and Data-Domain Deep LearningM. U. Ghani and W. C. Karl, “Deep Learning-Based Sinogram Completion for Low-Dose CT" IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), 2018, pp. 1-5.
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
18
Low-dose CT
Reduce dose by reducing number of projections
Full dose All projections
Reduced dose Sparse projections
Reduced dose = artifacts, noise Can be viewed as a sinogram completion problem!
But different structure than MAR
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
19
New Approach: Deep-LDCT
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
20
Sparse-view CT: 43 views Key points:
Learn complete sinogram behavior (vs patches) Use fully convolutional network (FCN) for scaling Generative Adversarial Network to avoid over-
smoothing Efficient (fast) and effective
Data-domain learning vs post-processing
Ghani, Muhammad Usman, and W. Clem Karl. "Deep Learning-Based Sinogram Completion for Low-Dose CT." 2018 IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP). IEEE, 2018.
Two steps:1. Conditional GAN (CGAN)
Sinogram Completion2. Filtered back projection
reconstruction
Generator Network
Discriminator NetworkReal/Fake?
CGAN
CGAN Network Architecture
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
21
x = sparse data y = complete data
Medical Images Database
2016 Low-Dose CT Grand Challenge [1] Parallel-beam data simulated for simplicity Poly-energetic CT simulation
Soft-segmentation [2] X-COM database [3]
1. C McCollough, “Tu-fg-207a-04: Overview of the low dose ct grand challenge,” Medical physics, vol. 43, no. 6Part35, pp. 3759–3760, 2016.2. Yiannis Kyriakou, Esther Meyer, Daniel Prell, and Marc Kachelrieß, “Empirical beam hardening correction (ebhc) for ct,” Medical physics, vol. 37, no. 10,
pp. 5179–5187, 2010.3. Martin J Berger and JH Hubbell. Xcom: Photon cross sections on a personal computer. Technical report, National Bureau of Standards, Washington, DC
(USA). Center for Radiation Research, 1987.
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
22
Detector channels: 720 Projection angles: 256 Attenuation properties used from XCOM database in 1
keV increments [1] Image dimensions: 300mm x 300mm Sparse-view data
Acquired at every 7th angle only 17% of complete data
Training data Data for 9 patients: 5,403 slices
Test data: 1,000 examples 3,000 pairs1. Martin J Berger and JH Hubbell. Xcom: Photon cross sections on a personal computer. Technical report, National Bureau of Standards, Washington, DC (USA).
Center for Radiation Research, 1987.
LDCT Dataset
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
23
CGAN Training
Tensorflow framework Stochastic gradient descent Batch size: 32 Adam optimizer with base learning rate: 0.0002 = 10 for loss One gradient step on the discriminator and one on the generator network. iterations on the discriminator for first 4 epochs. Trained for 500 epochs Training time : 48 hours on Nvidia Tesla P100 Testing time << 1 sec
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
24
Sinogram Completion Results and Residual
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
25Sparse-View Full-View CGAN Residual
Comparison to Dictionary Learning Based Sinogram Completion [1]
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
26Sparse-View Dictionary CGAN
Dictionary learning K-SVD Patch: 8x8 Dictionary: 64x256
Iterations: 30 Trained dictionary
used to in-paint the sparse-view data.
CGAN outperforms K-SVD in sinogram completion task.
1. Si Li, Qing Cao, Yang Chen, Yining Hu, Limin Luo, and Christine Toumoulin, “Dictionary learning based sinogram inpainting for ct sparse reconstruction,” Optik-International Journal for Light and Electron Optics, vol. 125, no. 12, pp. 2862–2867, 2014.
Comparison to TwIST Based Reconstruction [1]
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
271. Byung Gyu Chae and Sooyeul Lee, “Sparse-view ct image recovery using two-step iterative shrinkage-thresholding algorithm,” ETRI Journal, vol. 37, no. 6, pp. 1251–1258, 2015.
Sparse-View TwIST Deep-LDCT Full-view
Res
idua
l
Comparison to TwIST Based Reconstruction [1]
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
281. Byung Gyu Chae and Sooyeul Lee, “Sparse-view ct image recovery using two-step iterative shrinkage-thresholding algorithm,” ETRI Journal, vol. 37, no. 6, pp. 1251–1258, 2015.
Sparse-View TwIST Deep-LDCT Full-view
Res
idua
l
Quantitative Analysis
Comparison of considered methods
Deep LDCT provides good reconstructions efficiently.
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
29
Integrating Data and Image-Domain Deep LearningM. U. Ghani and W. C. Karl, “Integrating Data and Image Domain Deep Learning for Limited Angle Tomography using Consensus Equilibrium”, IEEE International Conference on Computer Vision Workshops (ICCVW), 2019.
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
30
Integrating Data and Image Domain Learning
Data-domain learning Fast and reasonable quality – but can we do better?
Image-domain learning in an MBIR framework Principled approach with excellent quality
Can we combine data-domain and image-domain learning in a unified framework?
Initial work based on consensus equilibrium
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
31
Image Reconstruction
Image Post-Processing
Data Pre-Processing
Raw Data Image
Integrated Data and Image Domain Learning
Image-domain DLData-domain DL
Consensus Equilibrium
Achieve equilibrium/compromise among multiple “agents”
Examples F1 = sensor fidelity, F2 = image prior Integrate multiple denoisers1. Distributed fast iterative reconstruction2. Allows integration of sensing models and image-domain learning.
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
32
CE solution
1. Buzzard, Gregery T., et al. "Plug-and-Play Unplugged: Optimization-Free Reconstruction Using Consensus Equilibrium." SIAM Journal on Imaging Sciences 11.3 (2018): 2001-2020.
2. Sridhar, Venkatesh, Gregery T. Buzzard, and Charles A. Bouman. "Distributed Framework for Fast Iterative CT Reconstruction from View-subsets." Electronic Imaging 2018.15 (2018): 102-1.
Integrated Data and Image-Domain Deep Learning via Consensus Equilibrium (DICE) Use Consensus Equilibrium Framework [1]
Multiple agents for information fusion
Consensus Equilibrium Agents
Data-domain learning in Image-domain learning in
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
33[1] G. T. Buzzard, S. H. Chan, S. Sreehari, and C. A. Bouman, “Plug-and-play unplugged: Optimization-free reconstruction using consensus equilibrium,” SIAM Journal on Imaging Sciences, vol. 11, no. 3, pp. 2001–2020, 2018.
Fdata (v1) argminx0
12 ycomplete Rx
W
2 1
2 2 v1 x2
2
Fimage (v2 ) image (v2 )
completey)( 2image v
Data-Domain and Image-Domain Learning via Consensus Equilibrium for Limited Angle CT
M. U. Ghani and W. C. Karl, “Integrating Data and Image Domain Deep Learning for Limited Angle Tomography using Consensus Equilibrium”, IEEE International Conference on Computer Vision Workshops (ICCVW), 2019.
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
34
Limited Angle CT
Security Systems with Non-rotational Scanning Imaging highly-dynamic scenes
Available Views: [00, 900] FBP requires complete coverage Image post-processing: fail to recover object features!
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
35
Available ViewsMissing Views
FBP
Sinogram
Data Domain Deep Learning via CGAN
( )
( )
( )
Skip Connections
1024x768 1024x76864128
256512
512512
1024
5121024
256128
Downsampled: 512x384x2
64128
256512
512512512
Generator Network ( )
Discriminator Network ( )
Conv, ELU
Conv, BN, ELUTransposed Conv, BN, ELUTranposed Conv
Fully Connected
( )
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
36
Image Domain Deep Learning via CGAN
( )
( )
( )
( )
64x64 64x6464
64x64x2
64
Generator Network ( )
Discriminator Network ( )
...
64 64 64
Total16 layers
6464
646432
Conv, LReLU
Conv, BN, LReLU
Fully Connected,BN, LReLU
Conv
Fully Connected
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
37Image-domain network cleans up image artifacts and noise
Dataset
ALERT TO4 Dataset [1] Image size: 475 mm x 475 mm Parallel re-binned data
Detectors: 1024 Limited Views: 360 uniform in [00, 900]
Training: 168 bags Testing: 21 bags Data DL trained on (incomplete,full) sinogram pairs Image DL trained on patches Ground truth images: Full–View MBIR [2]
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
38[1] Crawford, C, “Advances in automatic target recognition (ATR) for CT based object detection system–Final report, Dept,” Homeland Security Center Excellence, Task order number HSHQDC–12–J–00429, Boston, MA, 2014.[2] Jin, Pengchong and Bouman, Charles A and Sauer, Ken D, “A Model-Based Image Reconstruction Algorithm With Simultaneous Beam Hardening Correction for X-Ray CT.” IEEE Trans. Computational Imaging, vol. 1, no. 3, pp. 200–216, 2015.
Results -- Reconstructions
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
39
FBP + PP Data DL + FBP Plug-and-Play [1] DICE (Ours) Ground Truth
[1] D. H. Ye, S. Srivastava, J.-B. Thibault, K. Sauer, and C. Bouman, Deep Residual Learning for Model- Based Iterative CT Reconstruction using Plug-and-Play Framework," in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 6668{6672.
MBIR Image-domain Learning Only
Data-domain Learning Only
MBIR Integrated Data and Image-domain
Learning
Post-processing Image-domain Learning Only
Quantitative Comparison
Test dataset: 315 slices
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
40
Summary
Data-domain learning can achieve “near-optimal” performance by itself in some problems
Data-domain learning has a role in situations where maintaining conventional work flows matter Computation, time, speed, hardware investment, etc. matter
Data-domain learning can be combined with image-domain learning to achieve improved results!
Current work aimed at tighter integration Preliminary results show even more improvement
Integrating learning on both sides of the sensing operator divide should be a part of our toolkit
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
41
Thank You!
Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19
42
Top Related