High-Quality Hyperspectral Reconstruction Using a Spectral Prior

Inchang Choi† Daniel S. Jeon† Giljoo Nam† Min H. Kim† Diego Gutierrez*

†KAIST * Universidad de Zaragoza, I3A

High-Quality Hyperspectral Reconstruction Using a Spectral Prior

Light and Color Imaging

2

Bayer pattern RGB imaging Continuous spectra of light

Hyperspectral Imaging (HSI)

3

Compressive hyperspectral imaging Hyperspectral imaging

Scene spectra Projection

Reconstruction is an inverse problem of optical imaging

Shear Mask Unshear

Compressive Hyperspectral Imaging

4

Total variation Ground truth

5

Straightforward Approach

• Learning a regression function using a CNN

6

CNN for regression

Scene spectra Projection

The Regression Network Fails

7

regression ground truth

Hyperspectral Reconstruction

Encoder Decoder

Nonlinear representations

8

Hyperspectral reconstruction

Our Approach

9

Our reconstruction

10

Our reconstruction

Our reconstruction

11

Related Work -  Hyperspectral Imaging -  Compressive Hyperspectral Reconstruction

12

HSI without Reconstruction

13

Bandpass filter [Mansouri et al. 2007]

Pushbroom [Brusco et al. 2006]

LCTF (liquid crystal tunable filter)

[Attas et al. 2003]

HSI with Reconstruction

14

Scene Spectra Compressive image

Reconstruction

CASSI [Wagadarikar et al. 2008]

DD-CASSI [Gehm et al. 2007]

SS-CASSI [Lin et al. 2014]

[Jeon et al. 2016]

Image Formation

15

Shear Spectra Mask Unshear and project Observed image

Observation(2D)

Spectra(3D)

Lightmodulation(3Dto2D)

Hyperspectral Reconstruction

16

underdetermined system

# equations # unknowns ≪ 2

2

“Find a hyperspectral image that satisfies the image formation”

Reconstruction using TV-L1 Prior

17

1

𝛁𝒙𝒚 𝝉 2

2

TwIST [Bioucas-Dias and Figueiredo 2007]

SpaRSA [Wright et al. 2009]

-  TV-L1 is very common in computational photography

Reconstruction using Sparse Coding

18

2

2 1

- Use an overcomplete dictionary and a sparse code to represent a data

[Lin et al. 2014]

For all overlapping image patches

D: a dictionary

: a sparse code

Autoencoder -  For Our Deep Spectral Prior

19

Autoencoder

20

[Hinton and Salakhutdinov 2006]

x1

x2

x3

x4

y1

y2

y3

y4

Nonlinear Representation

21

x1

x2

x3

x4

y1

y2

y3

y4

Nonlinear representation

Autoencoder: Encoder and Decoder

22

x1

x2

x3

x4

y1

y2

y3

y4

Encoder : generate nonlinear representation

Decoder : produce data from representations

Nonlinear representation

Hyperspectral Reconstruction -  Learning a Spectral Prior -  Reconstruction with Alpha-fidelity

23

Overview of Our Reconstruction

24

Convolutional autoencoder

Hyperspectral image datasets

Reconstructed hyperspectral

image

Encoder / decoder

Optimization Compressive sensor input

Learning hyperspectral image prior

Hyperspectral reconstruction

Decoder

Autoencoder of Hyperspectral Images

25

Encoder Decoder

Convolutional autoencoder of hyperspectral images

Encoder

Nonlinear representation

Autoencoder of Hyperspectral Images

26

- 64 feature maps

- 3 x 3 convolution without pooling - ReLU activation function

64 64 64

Training Data

27

[Yasuma et al. 2006] Columbia dataset

[Chakrabarti and Zickler 2011] Harvard dataset

Validating Autoencoder

28

ground truth reconstruction (44.24 dB / 0.98)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

420 470 520 570 620 670

refle

ctan

ce

wavelength [nm]

GT gray patch red flower

Our Reconstruction - Data Term

29

Observation(2D)

Spectra(3D)

Lightmodulation(3Dto2D)

Shear Spectra Mask Unshear and project Observed image

Our Reconstruction - Data Term

30

Encoder Decoder

Nonlinearrepresentation

𝝰

Our Reconstruction

31

2

2

1

𝛁𝒙𝒚 𝝉

𝝰 

How can we utilize the encoder?

32

Decoder -  produce h (hyperspectral images)

from (nonlinear representations) -  a prior on h

-  know how h looks like

Encoder -  generate from h

-  know how looks like -  a prior on

Decoder

Encoder

Our Reconstruction with fidelity Prior

33

2

2

2

2

𝝉1 1

𝛁𝒙𝒚 𝝉2

“The nonlinear representation should be close to what the encoder knows.”

Impact of fidelity Prior

34

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

420 470 520 570 620 670

refle

ctan

ce

wavelength [nm]

GT

w/o 𝜶-prior

w/ 𝜶-prior

Yellow feather

Results -  Our Dataset -  Synthetic Results -  With a Real Compressive Imager

35

Our High-Quality Dataset

36

Download from http://vclab.kaist.ac.kr

Synthetic Result with Our High Quality Dataset

37

groundtruth

ours

Ourdataset

34.20dB / SSIM:0.95 32.03dB / SSIM:0.95 32.29dB / SSIM:0.92 39.21dB / SSIM:0.97

0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14

420 470 520 570 620 670

refle

ctan

ce GT

TwIST (0.006)

SpaRSA (0.010)

SC (0.019)

Ours (0.006)

TwIST[Kim2012]

SpaRSA[Wright2009]

sparsecoding[Lin2014]

38

groundtruth

TwIST[Kim2012]

SpaRSA[Wright2009]

sparsecoding[Lin2014] ours

28.42dB / SSIM:0.91 28.57dB / SSIM:0.92 34.89dB / SSIM:0.96 35.40dB / SSIM:0.97

Columbiadataset

0.00

0.05

0.10

0.15

0.20

0.25

0.30

420 470 520 570 620 670

refle

ctan

ce

GT

TwIST (0.058)

SpaRSA (0.062)

SC (0.039)

Ours (0.016)

Synthetic Result with Columbia Dataset [Yasuma et al. 2010]

39

CompressiveinputGroundtruthOurreconstruction

Synthetic Result with Our High Quality Dataset

Synthetic Result with Our High Quality Dataset

40

CompressiveinputGroundtruthOurreconstruction

Our DD-CASSI Result

41

camera coded aperture

prism prism objective lens

relay lens

imaging lens

[Gehm et al. 2007]

Our DD-CASSI Result

42

1

2

OurreconstructionCompressiveInput

0.0

0.2

0.4

0.6

0.8

1.0

420 470 520 570 620 670

GT Ours

0.0

0.2

0.4

0.6

0.8

1.0

420 470 520 570 620 670wavelength[nm]

1

2

Applications -  Spectral Interpolation -  Hyperspectral Demosaicing

43

Changing Modulation Matrix

44

2

2

2

2 1

for super-resolution: blurring + downsampling

Our reconstruction:

Note: the observation i should be modified accordingly

Spectral Interpolation

45

GroundTruth

Spectral Interpolation

46

16channels(52%)

Spectral Interpolation

47

8channels(26%)

Spectral Interpolation

48

3channels(10%)

49

Spectral Interpolation

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

420 470 520 570 620 670

refle

ctan

ce

wavelength [nm]

GT

52% (0.011)

26% (0.014)

10% (0.077)

16channels

8channels

3channels

GroundTruth

Hyperspectral Demosaicing

50

450nm 520nm

580nm 650nm

Bayer image

Hyperspectral Demosaicing

51

Ground truth

Hyperspectral Demosaicing

52

31.04dB / SSIM:0.89

0.00

0.10

0.20

0.30

0.40

420 470 520 570 620 670

refle

ctan

ce

wavelength [nm]

GT Bayer (0.020)

Demosaiced

Conclusion

53

Conclusion

• Learned a spectral prior using a convolutional autoencoder

• Proposed a novel hyperspectral reconstruction using the learned prior

• Demonstrated interesting applications

• Published a high quality hyperspectral dataset

54

Acknowledgments •  Seung-Hwan Back, Incheol Kim, Adrian Jarabo, and Paz Hernando

•  Min H. Kim acknowledges •  Korea NRF grants (2016R1A2B2013031, 2013M3A6A6073718) •  Giga Korea Project (GK17P0200) •  MCST •  Samsung Electronics (SRFC-IT1402-02) •  ICT R&D program of MSIT/IITP of Korea (R7116-16-1035)

•  Diego Gutierrez acknowledges •  ERC under EU’s Horizon 2020 research •  CHAMELEON project (682080) •  The Spanish Ministerio de Economia y Competitividad (TIN2016-78753-P)

55