2010/10/13 VCLAB, National Tsing Hua University, Taiwan 1

Multiview Video

Kai-Chao Yang

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 2

Outline

History of Video Coding Standards Definition of Multiview Video Applications of Multiview Video Concept of Stereo Video Multiview TV/Video System

Multiview Content Capture Correction Coding Display

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 3

History of Video Coding Standards

H.264/MPEG4 AVCH.262/MPEG-2

1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010

H.261 H.263 H.263+ H.263++

MPEG-1 MPEG-4 v2/visual

SVC MVC

ISOMPEG

JVT

ITU-TVCEG

FRExt

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 4

Definition of Multiview Video

Multiview video Multiple cameras are used to simultaneously acquire va

rious viewpoints of a scene.

Multiview video coding Encoding of sequences captured simultaneously from m

ultiple cameras using a single video stream.

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 5

Applications of Multiview Videos

Free viewpoint TV The viewers can experience the free viewpoint navigatio

n within the range covered by the cameras http://www.youtube.com/watch?v=0yP_J6M4fiU

Three-dimensional TV

Immersive teleconference

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 6

Examples (1/4)

Entertainments 3D game, FVV, …

Medicine 3D microscope, 3D endoscope, …

Education 3D model, 3D classroom, …

Surveillance Nurse, object recognition and tracking, …

Simulation Flight simulation, …

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 7

Examples (2/4)

3D microscope and 3D endoscope

http://www.njneurosurgeons.com/news/2010/04/19/new-jerseys-first-3d-endoscopic-surgery-performed-by-ninj-faculty.html

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 8

Examples (3/4)

Pedestrian Tracking

Kyungnam Kim and Larry S. Davis, "Multi-Camera Tracking and Segmentation of Occluded People on Ground Plane using Search-Guided Particle Filtering", European Conference on Computer Vision (ECCV), LNCS, 2006.

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 9

Examples (4/4)

Flight Simulation

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 10

Concept of Stereo Image/Video

3D information: Determination of relative depths in a perceived scene Stereopsis Accommodation of the eyeball Occlusion

Linear perspective

Vertical position

…

2D knowledge

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 11

Concept of Stereo Video (3D Video)

Stereopsis visual perception leading to the sensation of de

pth from the two slightly different projections of the world onto the retinas of the two eyes

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 12

Concept of Stereo Video

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 13

Multiview TV System

Content Delivery DisplayEncoding Decoding

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 14

Multiview Content

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 15

Multiview Content

Camera capture Stereo cameras Camera array

Computer generated Objects is defined Depth is inherent

Conversion from 2D video Detecting objects Assigning depth Filling occluded parts

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 16

Camera Capture

Stanford multi-camera array (128 video cameras)

Google street view camera (Image only)

Panasonic Full-HD 3D camera

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 18

Correction

Rectification of misalignment

Normalization of colors

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 19

Multiview Video Coding

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 20

Representation Formats (Full-Resolution)

Full-Resolution Stereo and Multiview Double data rate for stereo videos N-fold data rate for N-view videos

Efficient compression is the key issue MVC extension of H.264/AVC

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 21

Representation Formats (Stereo Interleaving) Stereo Interleaving

A multiplex of the two views into a single sequence Spatial multiplexing

The left and right views are sub-sampled and interleaved into a single frame

Temporal multiplexing The left and right views are interleaved as alternating frames

Advantages Compatible with existing codecs and delivery infrastructure

Drawbacks Loss of spatial or temporal resolution

RLRL

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 22

Representation Formats (Depth-based) Depth-based Format

2D video + depth map

Advantages Backward compatibility with the older coding standards Supporting both stereo and multiview displays Depth is adjustable

Drawbacks Limited depth range Occlusions Other views have to be synthesized

2D viewSynthesized

view

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 23

Multiview Video Coding

Single view

Multi-view Independent encoding

Low coding complexity but also low coding efficiency Inter-view prediction

Exploiting both spatial and temporal redundancy

I PB B B B B B P

I PB B B B B B P

I PB B B B B B P

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 24

Multiview Video Coding

Multiview camera

PB

BB

PB

BB

I

PB

BB

PB

BB

B

PB

BB

PB

BB

P

PB

BB

PB

BB

I

PB

BB

PB

BB

B

PB

BB

PB

BB

P

PB

BB

PB

BB

I

PB

BB

PB

BB

B

PB

BB

PB

BB

P

B

I B P

P

PB

P

P

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 25

View Generation

View 1 View 2synthesis

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 2626

View Generation

View synthesis Disparity-based

e.g. view1view3/2 view2 object(0,0) object(3,0) object(6,0)

Depth-based 2D + depth map

http://www.imec.be/ScientificReport/SR2007/html/1384302.html

DEPTH MAPS EXTRACTION FROM MULTI-VIEW VIDEOShttp://www.youtube.com/watch?v=KtRSbey1sKM

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 27

View Generation

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 28

Display (3D space)

Volumetric video

Holographic video

http://www.youtube.com/watch?v=kIDgC2no1uo

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 29

Display

Free viewpoint video

http://www.youtube.com/watch?v=vyhz8KgW49E http://www.youtube.com/watch?v=GSumx0Zs2XA

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 30

Display with 3D Glasses

Stereo video with glasses Left view and right view

Passive and active

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 31

Display with 3D Glasses

Anaglyph, polarized, and shutter glasses

http://www2.ciw.com.cn/h/2562/357866-17902.html

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 32

Display

Lenticular panel and Parallax barrier

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 33

Display

Stereo view display without glasses

A1A1A1A1A1A1A1A1A1A1

A2A2A2A2A2A2A2A2A2A2

Pixel 2

Pixel 1

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 34

Display

Multiview display without glasses

A2A2A2A2

A1A1A1A1B1B1B1B1C1C1C1C1

B2B2B2B2C2C2C2C2

Pixel 2

Pixel 1

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 35

Conclusions

An overview of multiview video from capture to display.

Issues Display

Reduction of brightness Reduction of refresh rate Reduction of temporal of spatial resolution

Coding efficiency Decoding delay 2D 3D View synthesis …

2010/10/13 VCLAB, National Tsing Hua University, Taiwan 36

Reference

A. Vetro, “Representation and Coding Formats for Stereo and Multiview Video,” Tech. Rep. TR2010-011, April, 2010.

S. Gaël, “Depth Map Estimation and Use for 3DTV,” Tech. Rep. n0379, February, 2010.

Y.-S. Ho and K.-J. Oh, “Overview of Multi-view Video Coding,” IWSSIP, 2007.

許精益 and 黃乙白 “ 3D 立體顯示技術之發展與研究 ,” 光學工程第 98 期 , 96 年 6 月 .