W. Clem Karl

M. Usman Ghani

Department of Electrical and Computer EngineeringDepartment of Biomedical EngineeringBoston University

Data and Image Domain Deep Learning for Tomographic Computational Imaging

Overview

Data and Image Domain Learning Tomography overview and some context

Some of Our Recent Work Data-domain Deep Learning

Metal artifact mitigation Sparse-view CT

Integrating Data Domain and Image Domain Learning Consensus Equilibrium Limited Angle CT application

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

2

Model: Goal make an image x from y Applications:

Security – a motivating example for us Medical Defense (e.g. SAR)

Computed Tomography

3

Physical SystemSinogram

DataImage

Rxy

x yR

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

Making Images with Learning

Image Domain vs Data Domain? Image Domain Learning: XX mappings Data Domain Learning: YY mappings

Most work uses Image Domain Learning Our focus today: Data Domain Learning

Good or bad idea? I hope to convince you it can be good!

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

4

R

Data DomainImage Domain

Image Reconstruction

Image Post-Processing

Data Pre-Processing

Raw Data Image

• Plug-and-Play• Proximal-like operators• Unrolled algorithms

• Image-domain DL• Data-domain DL

"" 1R

Case 1: Metal Artifact Reduction via Data-Domain Deep LearningM. U. Ghani and W. C. Karl, "Fast Enhanced CT Metal Artifact Reduction using Data Domain Deep Learning," in IEEE Transactions on Computational Imaging, 2019.

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

5

Applications with Metal Artifacts

Security

Medical Imaging

Esther Meyer, Rainer Raupach, Michael Lell, Bernhard Schmidt, and Marc Kachelrieß. Normalized metal artifact reduction (nmar) in computed tomography. Medical physics, 37(10):5482–5493, 2010.

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

Sinogram

Sinogram

6

CT Metal Artifact Reduction

Security Motivation Improve detection in X-ray-based CT scanners by reducing

metal artifacts Iterative reconstruction can reduce artifacts and improve

detection But… iterative reconstruction is slow, priors can be generic,

brittle

Idea: Use Data Domain Deep Learning Focus on pre-reconstruction correction of metal artifacts in

X-ray CT data Goal: A fast method within the standard workflow

“Bags per hour” matter Explore use of synthetic training data

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

7

Deep-MAR Framework Idea

Key points: Data-domain-focus – pose as problem of sinogram completion Learn complete sinogram behavior (vs patches) Use fully convolutional network (FCN) for efficiency Use C-GAN for good performance Synthetic data training, coupled with real data tuning

Speed vs Performance

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

8

Generator Network

Discriminator NetworkReal/Fake?

Conditional Generative Adversarial Network

Input Sinogram

FBP

Output Sinogram

Reconstruction without Metals

Deep-MAR Framework Details

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

9

Conditional Generative Adversarial Network (CGAN)

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

10

Generator Network

Discriminator NetworkReal/Fake?

Conditional Generative Adversarial Network

CGAN Data-Completion Network

Mean Squared Error (MSE) loss function over-smooths results in image-to-image tasks.

CGAN loss:

Generator loss:

y = incomplete raw data = complete data

Mask-specific data completion: loss focused on missing data

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

11

))](,(1[log(),(log(),( , yGyDEyyDDGL ycyycGAN c

])([),(maxminarg 2

2,* yGyDGLG cyycGANDG c

cy

Training Scanned bag data is limited

Perform majority of training on simulated data Fine tune with transfer learning on limited real data

Training only requires incomplete data pairs Material knowledge not required

Overall significant improvement vs training from scratch.

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

12

Example Dataset

Simulated Data Simulation setup based on Imatron C300 10,000 examples with up to 5 metallic objects 50,000 sinogram pairs

Transfer learning with real data 1,706 examples from ALERT TO4 data [1] at 130KeV 8,530 sinogram pairs

Images: 475 mm x 475 mm

Parallel beam re-binned data 1024 detectors 720 angular views

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

13[1] Crawford, C, “Advances in automatic target recognition (ATR) for CT based object detection system–Final report, Dept,” Homeland Security Center Excellence, Task order number HSHQDC–12–J–00429, Boston, MA, 2014.

Results – Simulated Example

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

14LI-MAR = Linear interpolation MARWNN-MAR = Weighted Nearest-Neighbor MARDeep-MAR = Deep Learning MAR

Input Target LI-MAR WNN-MAR Deep-MAR

Uncorrected LI-MAR WNN-MAR Deep-MAR

Results – Real Example

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

15LI-MAR = Linear interpolation MARWNN-MAR = Weighted Nearest-Neighbor MARDeep-MAR = Deep Learning MAR

Input Target LI-MAR WNN-MAR Deep-MAR

Uncorrected LI-MAR WNN-MAR Deep-MAR

Attention Maps

Occlude 11x11 patches with random U(0,max). Attention Maps: log-normalized resulting MSE Data near gaps most important Bigger gaps need more data Challenging problems use more of the sinogram – problem adaptive

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

16

Latent space analysis

Encoder representation t-SNE visualization [1]

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

17L. v. d. Maaten and G. Hinton, “Visualizing data using t-sne,” Journal of machine learning research, vol. 9, no. Nov, pp. 2579–2605, 2008.

Input projection data

Latent representation

Case 2: Low-Dose CT via Sparse-Projections and Data-Domain Deep LearningM. U. Ghani and W. C. Karl, “Deep Learning-Based Sinogram Completion for Low-Dose CT" IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), 2018, pp. 1-5.

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

18

Low-dose CT

Reduce dose by reducing number of projections

Full dose All projections

Reduced dose Sparse projections

Reduced dose = artifacts, noise Can be viewed as a sinogram completion problem!

But different structure than MAR

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

19

New Approach: Deep-LDCT

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

20

Sparse-view CT: 43 views Key points:

Learn complete sinogram behavior (vs patches) Use fully convolutional network (FCN) for scaling Generative Adversarial Network to avoid over-

smoothing Efficient (fast) and effective

Data-domain learning vs post-processing

Ghani, Muhammad Usman, and W. Clem Karl. "Deep Learning-Based Sinogram Completion for Low-Dose CT." 2018 IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP). IEEE, 2018.

Two steps:1. Conditional GAN (CGAN)

Sinogram Completion2. Filtered back projection

reconstruction

Generator Network

Discriminator NetworkReal/Fake?

CGAN

CGAN Network Architecture

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

21

x = sparse data y = complete data

Medical Images Database

2016 Low-Dose CT Grand Challenge [1] Parallel-beam data simulated for simplicity Poly-energetic CT simulation

Soft-segmentation [2] X-COM database [3]

1. C McCollough, “Tu-fg-207a-04: Overview of the low dose ct grand challenge,” Medical physics, vol. 43, no. 6Part35, pp. 3759–3760, 2016.2. Yiannis Kyriakou, Esther Meyer, Daniel Prell, and Marc Kachelrieß, “Empirical beam hardening correction (ebhc) for ct,” Medical physics, vol. 37, no. 10,

pp. 5179–5187, 2010.3. Martin J Berger and JH Hubbell. Xcom: Photon cross sections on a personal computer. Technical report, National Bureau of Standards, Washington, DC

(USA). Center for Radiation Research, 1987.

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

22

Detector channels: 720 Projection angles: 256 Attenuation properties used from XCOM database in 1

keV increments [1] Image dimensions: 300mm x 300mm Sparse-view data

Acquired at every 7th angle only 17% of complete data

Training data Data for 9 patients: 5,403 slices

Test data: 1,000 examples 3,000 pairs1. Martin J Berger and JH Hubbell. Xcom: Photon cross sections on a personal computer. Technical report, National Bureau of Standards, Washington, DC (USA).

Center for Radiation Research, 1987.

LDCT Dataset

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

23

CGAN Training

Tensorflow framework Stochastic gradient descent Batch size: 32 Adam optimizer with base learning rate: 0.0002 = 10 for loss One gradient step on the discriminator and one on the generator network. iterations on the discriminator for first 4 epochs. Trained for 500 epochs Training time : 48 hours on Nvidia Tesla P100 Testing time << 1 sec

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

24

Sinogram Completion Results and Residual

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

25Sparse-View Full-View CGAN Residual

Comparison to Dictionary Learning Based Sinogram Completion [1]

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

26Sparse-View Dictionary CGAN

Dictionary learning K-SVD Patch: 8x8 Dictionary: 64x256

Iterations: 30 Trained dictionary

used to in-paint the sparse-view data.

CGAN outperforms K-SVD in sinogram completion task.

1. Si Li, Qing Cao, Yang Chen, Yining Hu, Limin Luo, and Christine Toumoulin, “Dictionary learning based sinogram inpainting for ct sparse reconstruction,” Optik-International Journal for Light and Electron Optics, vol. 125, no. 12, pp. 2862–2867, 2014.

Comparison to TwIST Based Reconstruction [1]

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

271. Byung Gyu Chae and Sooyeul Lee, “Sparse-view ct image recovery using two-step iterative shrinkage-thresholding algorithm,” ETRI Journal, vol. 37, no. 6, pp. 1251–1258, 2015.

Sparse-View TwIST Deep-LDCT Full-view

Res

idua

l

Comparison to TwIST Based Reconstruction [1]

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

281. Byung Gyu Chae and Sooyeul Lee, “Sparse-view ct image recovery using two-step iterative shrinkage-thresholding algorithm,” ETRI Journal, vol. 37, no. 6, pp. 1251–1258, 2015.

Sparse-View TwIST Deep-LDCT Full-view

Res

idua

l

Quantitative Analysis

Comparison of considered methods

Deep LDCT provides good reconstructions efficiently.

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

29

Integrating Data and Image-Domain Deep LearningM. U. Ghani and W. C. Karl, “Integrating Data and Image Domain Deep Learning for Limited Angle Tomography using Consensus Equilibrium”, IEEE International Conference on Computer Vision Workshops (ICCVW), 2019.

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

30

Integrating Data and Image Domain Learning

Data-domain learning Fast and reasonable quality – but can we do better?

Image-domain learning in an MBIR framework Principled approach with excellent quality

Can we combine data-domain and image-domain learning in a unified framework?

Initial work based on consensus equilibrium

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

31

Image Reconstruction

Image Post-Processing

Data Pre-Processing

Raw Data Image

Integrated Data and Image Domain Learning

Image-domain DLData-domain DL

Consensus Equilibrium

Achieve equilibrium/compromise among multiple “agents”

Examples F1 = sensor fidelity, F2 = image prior Integrate multiple denoisers1. Distributed fast iterative reconstruction2. Allows integration of sensing models and image-domain learning.

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

32

CE solution

1. Buzzard, Gregery T., et al. "Plug-and-Play Unplugged: Optimization-Free Reconstruction Using Consensus Equilibrium." SIAM Journal on Imaging Sciences 11.3 (2018): 2001-2020.

2. Sridhar, Venkatesh, Gregery T. Buzzard, and Charles A. Bouman. "Distributed Framework for Fast Iterative CT Reconstruction from View-subsets." Electronic Imaging 2018.15 (2018): 102-1.

Integrated Data and Image-Domain Deep Learning via Consensus Equilibrium (DICE) Use Consensus Equilibrium Framework [1]

Multiple agents for information fusion

Consensus Equilibrium Agents

Data-domain learning in Image-domain learning in

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

33[1] G. T. Buzzard, S. H. Chan, S. Sreehari, and C. A. Bouman, “Plug-and-play unplugged: Optimization-free reconstruction using consensus equilibrium,” SIAM Journal on Imaging Sciences, vol. 11, no. 3, pp. 2001–2020, 2018.

Fdata (v1) argminx0

12 ycomplete Rx

W

2 1

2 2 v1 x2

2

Fimage (v2 ) image (v2 )

completey)( 2image v

Data-Domain and Image-Domain Learning via Consensus Equilibrium for Limited Angle CT

M. U. Ghani and W. C. Karl, “Integrating Data and Image Domain Deep Learning for Limited Angle Tomography using Consensus Equilibrium”, IEEE International Conference on Computer Vision Workshops (ICCVW), 2019.

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

34

Limited Angle CT

Security Systems with Non-rotational Scanning Imaging highly-dynamic scenes

Available Views: [00, 900] FBP requires complete coverage Image post-processing: fail to recover object features!

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

35

Available ViewsMissing Views

FBP

Sinogram

Data Domain Deep Learning via CGAN

( )

( )

( )

Skip Connections

1024x768 1024x76864128

256512

512512

1024

5121024

256128

Downsampled: 512x384x2

64128

256512

512512512

Generator Network ( )

Discriminator Network ( )

Conv, ELU

Conv, BN, ELUTransposed Conv, BN, ELUTranposed Conv

Fully Connected

( )

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

36

Image Domain Deep Learning via CGAN

( )

( )

( )

( )

64x64 64x6464

64x64x2

64

Generator Network ( )

Discriminator Network ( )

...

64 64 64

Total16 layers

6464

646432

Conv, LReLU

Conv, BN, LReLU

Fully Connected,BN, LReLU

Conv

Fully Connected

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

37Image-domain network cleans up image artifacts and noise

Dataset

ALERT TO4 Dataset [1] Image size: 475 mm x 475 mm Parallel re-binned data

Detectors: 1024 Limited Views: 360 uniform in [00, 900]

Training: 168 bags Testing: 21 bags Data DL trained on (incomplete,full) sinogram pairs Image DL trained on patches Ground truth images: Full–View MBIR [2]

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

38[1] Crawford, C, “Advances in automatic target recognition (ATR) for CT based object detection system–Final report, Dept,” Homeland Security Center Excellence, Task order number HSHQDC–12–J–00429, Boston, MA, 2014.[2] Jin, Pengchong and Bouman, Charles A and Sauer, Ken D, “A Model-Based Image Reconstruction Algorithm With Simultaneous Beam Hardening Correction for X-Ray CT.” IEEE Trans. Computational Imaging, vol. 1, no. 3, pp. 200–216, 2015.

Results -- Reconstructions

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

39

FBP + PP Data DL + FBP Plug-and-Play [1] DICE (Ours) Ground Truth

[1] D. H. Ye, S. Srivastava, J.-B. Thibault, K. Sauer, and C. Bouman, Deep Residual Learning for Model- Based Iterative CT Reconstruction using Plug-and-Play Framework," in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 6668{6672.

MBIR Image-domain Learning Only

Data-domain Learning Only

MBIR Integrated Data and Image-domain

Learning

Post-processing Image-domain Learning Only

Quantitative Comparison

Test dataset: 315 slices

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

40

Summary

Data-domain learning can achieve “near-optimal” performance by itself in some problems

Data-domain learning has a role in situations where maintaining conventional work flows matter Computation, time, speed, hardware investment, etc. matter

Data-domain learning can be combined with image-domain learning to achieve improved results!

Current work aimed at tighter integration Preliminary results show even more improvement

Integrating learning on both sides of the sensing operator divide should be a part of our toolkit

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

41

Thank You!

Data and Image Domain Deep Learning for Tomographic Computational Imaging 10/16/19

42