Light Field Compression Using 2-D Warping and Block Matching

EE398A - Compression of Light Fields using 2-D Warping and Block Matching

1

Light Field Compression Using 2-D Warping and Block Matching

Shinjini KunduAnand Kamat Tarcar

EE398A Final Project


2

Outline

• Motivation and Goals• Overview of Our Method• Results and Analysis• Summary• Future Work• References


3

Motivation

• Light field images are used in computer graphics to compute new views of a scene without need for scene geometry model1.

• Need to compress large set of images• Exploit inter-view coherence to achieve

compression.

1. M. Levoy and P. Hanrahan, “Light field rendering,” in Computer Graphics (Proceedings SIGGRAPH 96), August 1996, pp. 31-42.

Light Fields• Represents a 3D scene or object from all viewing

positions and directions– 2D array of 2D images– Difficult to Acquire– Very Large

• Perfect representation requires images of the order of the resolution


5

Light Field Views


6

Light Field Data Set8.4 MB uncompressed data setshttp://lightfield.stanford.edu/aperture.swf?lightfield=data/lego_lf/preview.zip&zoom=1

Credit: Andrew Adams

http://lightfield.stanford.edu/aperture.swf?lightfield=data/lego_lf/preview.zip&zoom=1


7

Related Work• Intra-frame coding

– Vector quantization, DCT coding, transform coding yield compression ratios of less than 30:1

• Inter-frame coding (compression in the hundreds, thousands)– Disparity compensation– 3D geometry models– Blockwise

Compression ideal: maximally use coherence between two images


8

Our Method: 2-D Warping

• Each consecutive view is a projection of the previous view due to constant predictable movement of camera

• Find this relation between the views by obtaining projection matrix for each pair of views

• Predict the view and encode the residual


9

Our Encoding Scheme

Reconstructed Previous View

Previous Frame 2-D Warped

Lagrangian Cost

Function

Cost=R1+λD1

Cost=R2+λD2

2D Warping Algorithm

2-D DCT for the Residual

Residual and MV

?

Input View

--

Use for Reconstruction


10

Notes• DCT used on 8x8 blocks to encode residual• Laplacian distribution assumed for motion vectors• Projection matrix was encoded by normalizing values with

respect to 10, and assuming Laplacian distribution of bitrate. The min and max values are encoded separately using binary encoding.

• H =

-0.578 0.005 -0.720 -0.003 -0.572 0.007 0.000 0.000 -0.582


11

1. Feature match by correlation2. Projective matrix computedLagrangian Mode Decision using two references3. Clipped edges are interpolated using motion

compensation


12

Getting a predicted projection:Step 1: Feature matching by Correlation

corners detected corners detected

Features detected by Harris corner detection algorithm, and matching points identified by maximum correlation


13

Computing the Homography Matrix

• A homography is an invertible transformation from the real projective plane to the projective plane that maps straight lines to straight lines


14

Results for 2-D Projection Warping

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 131

32

33

34

35

36

37

38

Rate, in bits/pixel

PS

NR

(dB

)

Lego Men

motion compensation + DCT onlyprojection method


15

Results for 2-D Projective Warping

0 0.5 1 1.5 2 2.529.5

30

30.5

31

31.5

32Crystal Ball

Rate, in bits/pixel

PS

NR

(dB

)

data is for crystal ball light fieldmotion compensation + DCT onlyprojection method


16

Results for 2D Projective Warping

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.529

30

31

32

33

34

35

36

37

38Lego Men vs. Crystal

PS

NR

(dB

)

Rate, in bits/pixel

Lego MenCrystal Ball


17

Compression Ratios

1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 60

50

100

150

200

250

300

350

400

450Compression Ratios

quantizer step size (log(Q))

com

pres

sion

ratio

projective methodmotion compensation only


18

Conclusion

• Advantages: decreased coding complexity, and increased rate/PSNR as well as compression

• Experimental results demonstrate improved coding efficiency with our 2D warp method when compared with MVC.


19

Future Work Possible

• Optimize the code to give better PSNR values and check performance by introducing extra modes like copy mode

• Explore other methods of using inter-view redundancy in detail like disparity compensation at sub-pel accuracy

• Run for larger data sets and optimize complexity of the algorithm


20

Summary

• Light fields represent a 3D scene using sequence of 2-D images

• Large amounts of data• Can use redundancy between images using 2-

D warping with motion compensated block matching

• Results in a sleek method for compression• Performance wise..


21

Acknowledgement

• Prof. Girod for pointing us in the right direction• Mina Makar for his help• Chuo-Ling Chang for DAPBT code• Huizhong Chen and Derek Pang for their help• Prof. Peter Kovesi for open source matlab

function library• Prof. Levoy’s group and Andrew Adams for

access to light field images


22

Questions?


23

Other Projects

• Use Motion Compensation with Directional Transforms

o Result: Gain in PSNR due to directionality is approximately 0.1dB at high Quantization; almost nil increase seen at low quantization

• So, We adapted the direction of out project to study a new approach of compression presented next.


24

Results with Motion Compensation and DAPBT for Crystal light field


25

Results with Motion Compensation and DAPBT for Lego light field


26

This is how blocking is done and direction selection happens!IAP(DAT)+IRP(DCT) for QP=44, Crystal Light Field


27

For Lego light field IAP(DAT)+IRP(DCT) for QP=44, Crystal Light Field

Light Field Compression Using 2-D Warping and Block Matching

Documents

Transcript of Light Field Compression Using 2-D Warping and Block Matching