An efficient method for Multi-Dimensional Compressive Imaging

Yair Rivenson and Adrian Stern Ben Gurion University of the Negev, Israel

Sampling Encoding

Conventional imaging:

N K K NDecoding

N K<<N

Q: So why acquire so much pixels and than throw awa y information?

Q: Is it possible to acquire only K pixels?

Compressed (Compressive ) imaging:

DecodingM M N

M~KEncoding +Sampling

fΦΦΦΦ g

N (=n2) Object pixels

M<N pixels

N pixels but only K nonzero K nonzeros

ΨΨΨΨTα

αααα-estimation ΨΨΨΨα f

N pixels

3

Image acquisition Digital image reconstructionObject representation

1min || || subject toα

α g =ΦΨα =Ωα

Multi-Dimensional Compressive Imaging

ObjectCoefficients

NRf ∈NR∈αΨ NRg∈

Measured image

Imaging system Object representation in terms of

NNR ×∈Φ

Imaging system Object representation in terms of compression coefficients

1−Ψ

NNR ×∈Ψ

NRg ∈

N

N

NR∈α

ΨΨΨΨΦ

NNR ×∈Φ

N

N

4Multi-Dimensional Compressive Imaging

Object representation in terms of

ObjectCoefficients

NRf ∈NR∈αΨ MRg∈

Measured image

Imaging system

NMR ×∈Φ

Object representation in terms of compression coefficients

Imaging system

1−Ψ

NNR ×∈Ψ

NRg ∈

N

N

NR∈α

ΨΨΨΨMΦ

NMR ×∈Φ

M

N

M<<N pixels are captured

5Multi-Dimensional Compressive Imaging

Number of samples M required to sample a K-sparse signal of length N:

( )2 , log( )M C K Nµ≥ ⋅ ⋅ ⋅Φ Ψ

Where µµµµ denotes the mutual coherence between ΨΨΨΨ and ΦΦΦΦ defined:

6 Multi-Dimensional Compressive Imaging

NN

1−Ψ

NNR ×∈Ψ

NRg ∈

N

NR∈α

ΨΨΨΨΦ

NMR ×∈Φ

M N

NN

1−Ψ

NNR ×∈Ψ

NRg ∈

N

NR∈α

ΨΨΨΨ

Φ

NMR ×∈Φ

M NHuge!

7 Multi-Dimensional Compressive Imaging

R∈Ψ R∈α

In imaging N~O(106) M~O(105)The sensing operator Φ has ~1011 independent entries

This creates sever implementation challenges:1. Sensing matrix storage and computational burden2. Optical implementation3. Optical System calibration

NN

1−Ψ

NNR ×∈Ψ

NRg ∈

N

NR∈α

ΨΨΨΨ

Φ

NMR ×∈Φ

M NHuge!

Since Φ is in the order of O(M·N)=O(KlogN·N) , even for highly compressible

image (e.g., 1:50) the sensing matrix for megapixel images

would require a storage place (in double precision) of 0.95 Terabyte!

8

3 6115 10 110⋅ × ⋅∈ℜΦ

Multi-Dimensional Compressive Imaging

R∈Ψ R∈α

Let’s take for example Rice’s single pixel imager

NN

Let us assume that for every random projection we need 5ms, then we will need115,000x5ms which is justabout 10 minutes to capture a single frame.

9Multi-Dimensional Compressive Imaging

NN

1−Ψ

NNR ×∈Ψ

NRg ∈

N

NR∈α

ΨΨΨΨ

Φ

NMR ×∈Φ

M N

Object plane

Image plane

Let us take for example, the single shot compressiv e sensing technique using a random phase mask 1

In order to determine the sensing operator,Φ, we needto measure N≈106

PSFs (for example , by shine with a point source for everyN≈106 objectpixels)

10

z1z2

Incoherent (random) projection element 1

1. A. Stern and B. Javidi, “Random projections imaging with extended space-bandwidth product”, IEEE/OSA journal of Display Technology, 3(3), 315-320, (. 2007)

Multi-Dimensional Compressive Imaging

Design separable1 sensing systems [ i.e. psf(x,y)=psf(x) psf(y)]

N

Φx

Φy

Multi-Dimensional Compressive Imaging 11

√ MΦM

11 1

1

. . .

. . .

. . .

. . .

. . .

n

m mn

a a

a a

⊗ =

B B

A B

B B

mp nqC ×⊗ ∈A B

The Kronecker product between two arbitrary matrices , is defined as:p qC ×∈Bm nC ×∈A

Φx Φy

N

√ N√ N

1. Y. Rivenson and A. Stern, IEEE Signal Processing Letters, 16(6), 449-452 , June 2009.

M<N pixels K nonzeros

A-estimation A

N pixels

xΦTyΦ

pixels but only K nonzero

AxΨ

TyΨ

FTxΨ yΨ

Object pixels

N x N

G

N x N

12

Digital image reconstruction

Object representation

( )1min

Tm T mx ysubject to =

α

Α G Φ ΨΑΨ Φ(Which is equivalent to the 2D formulation: )

Image acquisition

Multi-Dimensional Compressive Imaging

Reconstruction of4096x4096≈16Megapixel Shepp-Logan. (a) original image (b)

Perfect reconstruction fromM=860x860samples (~23:1), whenK=68,000 non-

zero terms under Haar wavelet transform.

Φ is determined by28 Megabyte for separable CS in contrast to73 Terabyte

(predicted) for conventional CS.13

(a) (b)

Multi-Dimensional Compressive Imaging

Recall that in order to assure ℓ1 norm minimization we need

samples.( )2 , log( )M C K Nµ≥ ⋅ ⋅ ⋅Φ Ψ

14 Multi-Dimensional Compressive Imaging

The number of samples required to assure ℓ1-norm minimization for

separable sensing operator is1:

( )4 1 1, log( )D DM C K Nµ≥ ⋅ ⋅ ⋅Φ Ψ

1. Y. Rivenson and A. Stern, IEEE Signal Processing Letters, 16(6), 449-452 , June 2009.

5000

6000

Predicted Separable CS

( )logPM M N=

21 11010

10

10 10

4log ( )2 log ( )( , )log ( )

( , ) 2 log ( ) 4 log ( )

D D S

S

nnn

N n

µµ⊗

≈ = =Φ Φ Ψ

Φ Ψ

Oversampling ratio ( # samples with separable CI / # samples with non separable CI), assuming Gaussian random projecetions

Meaning: log(n)=3 times more samples for a megapixel image!

15

0 1000 2000 3000 4000 5000 6000 7000 8000 90000

1000

2000

3000

4000

N

M(N

)

Conventional CS – MSamples

Separable CS – Ms Samples

1.25M

Multi-Dimensional Compressive Imaging

Empirically1, the oversampling factor is even smaller (~1.25)

1.Y. Rivenson and A. Stern, "Practical compressive sensing of large images", 16th International Conference on Digital Signal Processing (DSP 2009), July 2009

We saw the effectiveness of designing separable system 2Doperators (Cartesian x-y image representation)

What happens in r-Dimensions ? What happens if we use more than 2D separable operators?

Multi-Dimensional Compressive Imaging 16

What happens if we use more than 2D separable operators?

Motivation: •Many optical systems acquire multiple dimensional images (3D images, multispectral, video sequences)•Multi dimensional data is typically highly compressive (K<<N)

Using the relation:

1 2 1 2 1 1 2 2 1( ... , ... ) ( , ) ( ... , ... ) ( , )

r

r r r r i ii

µ µ µ µ=

⊗ ⊗ ⊗ ⊗ ⊗ ⊗ = ⊗ ⊗ ⊗ ⊗ = ΠΦ Φ Φ Ψ Ψ Ψ Φ Ψ Φ Φ Ψ Ψ Φ Ψ

It can be shown that for CI system with imaging ope rators separable in r-Dimensions the number of sample limit is

Multi-Dimensional Compressive Imaging 17

Oversampling factor (assuming Gaussian Random proje ctions)

For example:•We want to use CI for 3-D images, composed of 100 image projection (slices), each of 1 Megapixel. Let us assume that the compressibility of the 3D image is k/N= 200:1.

With conventional 3-D CI the storing matrix will take approximately 200 Terabyte, and calibration of the device is

Multi-Dimensional Compressive Imaging 18

approximately 200 Terabyte, and calibration of the device is merely impossible. However, if we decompose the sensing operator to r=3 separable operators storage requirements drop to approximately 3.5 Megabyte only!

The price to be paid is in taking (theoretically) 9 times more samples in order to guarantee recovery. In practice the oversampling factor is always much lower, estimated to be smaller than 3.

We have introduces the concept of usingr-dimensional separable sensingoperatorsfor Compressed Imaging.

Generally speaking, thisreduces the complexityfrom to

The main advantages of using r-dimensional separable sensing operatorsare: Reducessensorcalibrationtime, Reducessensorcalibrationtime,

Enables design and optimization of extremely high dimensional sensingmechanism,

Universal sensing operator.

It is shown that applying this method requires only a reasonable amount of additional samples

19 Multi-Dimensional Compressive Imaging

Any questions?Any questions?

Conventional CS – Number of samples –O(KlogN)

Matrix storage –O(KlogN · N)

Sensor calibration steps –O(N)

Toeplitz CS1 – Toeplitz CS1 – Number of samples –

Matrix storage –

Sensor calibration steps –

Separable CS – Number of samples –

Matrix storage –

Sensor calibration steps –

21

( log )O K N

( log )O K N N⋅

( )O N

2( log )O K N

( )O N(1)O

1Bajwa et al., “Toeplitz-structured compressed sensing matrices,”IEEE Workshop on Statistical Signal Processing,Madison, Wisconsin, August 2007 Multi-Dimensional Compressive Imaging

The entire design process requires an access to the Fourier domain, thus, making the design process not universal, and not suitable for every imaging application.

The design requires two sets of permutations, first in the Fourier domain, and then a randomized sign sequence or random placing in space of single pixels, i.e. random sampling of the Toeplitz in space of single pixels, i.e. random sampling of the Toeplitz matrix. In our approach we don't need the random sampling in the sensor domain, it's built-in the operator.

It is not clear whether convolution approach can be broaden for extra dimensions, i.e., for 3-D imaging.

Multi-Dimensional Compressive Imaging 22