Post on 06-May-2015
Fraunhofer Institute for TelecommunicationsHeinrich-Hertz-Institut Berlin Einsteinufer 3710587 BerlinGermany
3D Video: From Stereo to Multi-View
+49 30 310 02 – 0info@hhi.fraunhofer.de http://www.hhi.fraunhofer.de
Karsten Müller
TEWI - Kolloquium,Universität Klagenfurt
Kars
ten
Mül
ler
Overview
• Introduction, scenarios, requirements
• 3D Video: Stereo Video + MVC
• 3D Video: 3D Video Formats & Coding
• International 3D media research and development
• Summary
18.03.2010 Universität Klagenfurt
Slide 2
Kars
ten
Mül
ler
3D Video Introduction
• Provision of left and right eye view• Added depth sensation • Resolved 2D viewing
ambiguity• Additional features
(e.g. free viewpoint, depth-controlled object insertion)
3D Video is more than adding 1 dimension to 2D!
Slide 3
18.03.2010 Universität Klagenfurt
3D Video Scenarios3D cinema• Increasing number of 3D productions• First studios start to release all productions also in 3D• Technology: stereoscopic 3D, glasses based
3D home entertainment• Different types of displays available: stereoscopic, auto-stereoscopic
with 2 … N views• Various technologies, input formats and display sizes• Glasses based systems may not be acceptable
3D mobile• auto-stereoscopic 2 view display with fixed viewing position• Good 3D viewing in spite of small display sizes
Kars
ten
Mül
ler
Slide 4
18.03.2010 Universität Klagenfurt
18.03.2010© Fraunhofer HHI
Folie 5
Kars
ten
Mül
ler
Stereo Video Solution
• Original Cameras = Views for Display
• Production format specification (e.g. SMPTE)
• Rectification, color correction, format conversion
• Conventional Stereo Format, e.g. Side-by-side
• Multi-view video coding (MPEG-4 MVC/H.264)
• High-resolution glasses-based stereo display
Stereo-Capture
Post-processing
MVC-Coding
MVC-Decoding
ProductionFormat
TransportFormat
Transport Format
Stereo-Display
Display Format
3D format: Conventional Stereo video (CSV)Ka
rste
n M
ülle
r
Slide 6
18.03.2010 Universität Klagenfurt
Original Image Source: Interactive Visual Media Groupof Microsoft Research
Kars
ten
Mül
ler
Current Coding Standards for Stereo Video
• Advanced Video Coding (MPEG-4 AVC / H.264)– Simulcast– Stereo SEI (Supplementary Enhancement Information)– Auxiliary Picture Syntax
• MPEG-C part 3 (container format for V+D, e.g. with AVC coding)
• Multi-view Video Coding (MPEG-4 MVC / H.264)
Slide 7
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
Multi-view Video Coding
• Joint Coding of camera views with similar content
Slide 8
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
MVC Coding Method
• Coding approach for exploiting temporal and inter-view dependencies using MPEG4-AVC/H.264
• Usage of hierarchical B pictures in temporal direction
• Usage of P or hierarchical B pictures in inter-view direction
• Frame reordering to optimize memory usage
No H.264 changes required for multi-view coding
• Standardized in July 2008• Adopted for 3D Blu Ray in Dec. 2009
Slide 9
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
MVC Coding Structure- Picture Reordering
Slide 10
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
MVC Coding Structure
Slide 11
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
MVC Camera Setting Adjustment - Star
Slide 12
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
MVC Camera Setting Adjustment – 2D Array
Slide 13
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
MVC Coding Results
• Coding structure allows AVC coder, to select the best inter-view-temporal neighbors within the MVC sequence
• Coding gain obtained by usage of hierarchical B-frames and by exploitation of inter-view dependencies
Slide 14
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
MVC Coding Results – delta_PSNR
Slide 15
18.03.2010 Universität Klagenfurt
Broadcasters currently concentrate on stereo and will not invest in new infrastructuresThey will adopt a system, which they can feed through existing pipes: e.g. anaglyph or multiplexed stereo:
checkerboardside-by-sideabove-below etc.
Will Stereo TV be the long term solution?
anaglyph polarized shutter
Kars
ten
Mül
ler
Slide 16
18.03.2010 Universität Klagenfurt
Short Term 3DTV will be Stereo TV
The Answer is Given by Our ChancellorKa
rste
n M
ülle
r
Slide 17
18.03.2010 Universität Klagenfurt
3DTV viewing conditions are more complex than in 3D cinema:• Glasses are generally not accepted in the living room • There are many different 3D display technologies • Varying viewing conditions (distance, display size)• User preferences have to be considered• Not all users can see stereo (≈10%)• Not all users want to see stereo• Users might not want to see all programmes in stereo
Kars
ten
Mül
ler
Slide 18
18.03.2010 Universität Klagenfurt
Special Requirements for 3DTV
Kars
ten
Mül
ler
Multi-view Displays
• The most common form of multi-view displays utilise either lenticularscreens or parallax barriers.
• Lenticular screens consist of a series of vertically aligned or slanted cylindrical lenses and parallax barriers vertically aligned apertures.
• Light is guided in the appropriate directions by either focussing it or by blocking unwanted rays.
VIEW 1 VIEW 2 VIEW 3
LENTICULAR
VIEW 1 VIEW 2 VIEW 3
PARALLAX BARRIER23 1
1 4 44 11
SCREENSlide 19
18.03.2010 Image Source: De Montfort University, UK
Kars
ten
Mül
ler
3D Video Color only: Assumption 1
• Multi-view Displays require N views (e.g. 5,9,22,50)• Why not transmitting N color views with MVC?
Slide 20
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
MVV and MVC Restrictions
• Pure coding of multi-view video
• Still linear dependency of bit rate from number of views
• Number and position of views fixed over the whole 3D chain
• No baseline adaptation for different displays
• Number of Views fixed in MVV/MVC
Slide 21
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
MVC Restrictions
• Coding experiments on camera density
Original camera distance
Slide 22
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
MVC Restrictions (2)
• Coding experiments on camera density with Rena test sequence
average rate average rate per cameraSlide 23
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
3D Video Color only: Assumption 2
• Multi-view Displays require N views (e.g. 5,9,22,50)• Why not transmitting 2 or 3 color views with MVC
and reconstructing N views at the receiver?
Slide 24
18.03.2010 Universität Klagenfurt
Color Only Intermediate ViewKa
rste
n M
ülle
r
Slide 25
18.03.2010
Original Image Source: Interactive Visual Media Group of Microsoft Research
Depth mapKa
rste
n M
ülle
r
Slide 26
Color information 256 depth layers
18.03.2010 Universität Klagenfurt
Original Image Source: Interactive Visual Media Groupof Microsoft Research
Example Reconstruction
Pure image blending Depth image basedrendering
Original Image Source: Interactive Visual Media Groupof Microsoft Research
Kars
ten
Mül
ler
Slide 27
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
Advanced 3D Video Solutions Required
• Color-only multi-view video too restricted– High number of views cannot be efficiently
compressed and transmitted– View synthesis of new views at the receiver
side from color-only data gives badreconstruction quality
• Additional Scene Geometry information required, e.g. per-pixel depth/disparity data
18.03.2010 Universität Klagenfurt
Slide 28
18.03.2010© Fraunhofer HHI
Folie 29
Kars
ten
Mül
ler
Depth-enhanced 3D Video
• Stereo / multiview capturing, depth provision
• Production format specification (e.g. SMPTE)
• Rectification, color correction, format conversion
• 3D format specification
• 3D video coding (e.g. MPEG)
• High quality intermediate view synthesis (e.g. 2 view + 2 depth N view)
• High-resolution stereo/multiview display
Capture
Post-processing
Coding
View Synthesis
ProductionFormat
TransportFormat
Transport Format
Display
Display Format
Kars
ten
Mül
ler
3D Video Formats
• Conventional stereo video (CSV)• Mixed resolution stereo (MRS)• Video plus depth (V+D)
• Multiview video (MVV)• Multiview video + depth (MVD)• Layered depth video (LDV)
• Depth-enhanced stereo (DES)
Slide 30
18.03.2010 Universität Klagenfurt
assuming multiple views with color & depth… Ka
rste
n M
ülle
r
Slide 31
18.03.2010 Universität Klagenfurt
Stereo video (CSV)
CSV
Kars
ten
Mül
ler
Slide 32
18.03.2010 Universität Klagenfurt
Mixed resolution stereo (MRS)
MRS
Kars
ten
Mül
ler
Slide 33
18.03.2010 Universität Klagenfurt
Video plus depth (V+D)
V+D
Kars
ten
Mül
ler
Slide 34
18.03.2010 Universität Klagenfurt
MVV
Multiview video (MVV)Ka
rste
n M
ülle
r
Slide 35
18.03.2010 Universität Klagenfurt
MVD
Multiview video plus depth (MVD)Ka
rste
n M
ülle
r
Slide 36
18.03.2010 Universität Klagenfurt
LDV
Layered depth video (LDV)Ka
rste
n M
ülle
r
Slide 37
18.03.2010 Universität Klagenfurt
DES
Depth-enhance Stereo (DES)Ka
rste
n M
ülle
r
Slide 38
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
Towards a new 3D Video Coding Standard
• MPEG develops a new 3D video coding standard
• Motivation for new standard:– Decouple production from coding format– MPEG-4 AVC/H.264 only optimized for 2D color video, but not for
depth information
Slide 39
18.03.2010 Universität Klagenfurt
Advanced 3DTV concept based on MVDKa
rste
n M
ülle
r
Slide 40
18.03.2010 Universität Klagenfurt
3D components under consideration:– Transport format– 3D video coding methods– Intermediate view synthesis
18.03.2010© Fraunhofer HHI
Folie 41
Kars
ten
Mül
ler
Coverage of 3D Video Coding
Capture
Post-processing
Coding
View Synthesis
ProductionFormat
Transport Format
Transport Format
Display
Display Format
Kars
ten
Mül
ler
Challenges for 3DVC
• Consider capturing technology, i.e. maximal 2-3 recorded views
• Break linear dependency of coding bit rate from number of target views (e.g. MVC)
• Provide scene geometry data in general form, i.e. pixel-wise depth data
• Consider statistical properties of depth (and supplementary) data
• Consider new quality evaluation methods for intermediate views
• Provide high-quality view synthesis for continuous viewing range
• Decrease depth and coding errors
Slide 42
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
Coding Experiments MVD with MVC
• Joint color and depth coding for optimal intermediate view quality
• Evaluation of total bit rate (e.g. of 2 color and 2 depth maps) vs. quality of decoded synthesized views
• View synthesis is part of coding optimization
• Reduction of coding and interpolation artifacts through high-quality view synthesis
Slide 43
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
Coding Results Intermediate View
• Ballet (JMVM 7.0.1, GOP16, κ=½)
Slide 44
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
Coding Results Intermediate View
• Breakdancers (JMVM 7.0.1, GOP16, κ=½)
Slide 45
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
Color Depth Bit Rate Distribution
• Objective and subjective examples for optimal and non-optimal bit rate distribution
• Variation of color and depth quality
• Tradeoff between synthesized views and views at original positions
Slide 46
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
Coding Results Intermediate View
• Ballet (JMVM 7.0.1, GOP16, κ=½)
C30D30
C24D40
Slide 47
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
Coding Results across Viewing Range
• Ballet (JMVM 7.0.1, GOP16)
Slide 48
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
Coding Results Intermediate View
• Breakdancers (JMVM 7.0.1, GOP16, κ=½)
C30D36
C36D30
Slide 49
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
Coding Results across Viewing Range
• Breakdancers (JMVM 7.0.1, GOP16)
Slide 50
18.03.2010 Universität Klagenfurt
View Synthesis for MVD
– High quality view interpolationis essential for 3D video applications with MVD data.
– Processing steps:- Layer Extraction- Layer Projection- Hole Filling + Filtering
Kars
ten
Mül
ler
Slide 51
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
Synthesis Improvements (Ballet)
compressed uncompressed
simple merging
layer-basedview synthesis
Slide 52
18.03.2010 Universität Klagenfurt
International R&D in 3D Media
• ISO/IEC MPEG 3DVC (3D video coding)• SMPTE Task Force of 3D to the home (3D master format)• ITU-R SG6• DVB – TM - ES Stereoscopic (3D) TV• 3D@Home Consortium• Association in Korea, Japan,…• EU 3D, Immersive, Interactive Media Cluster
• International Conferences– 3DTV-CON– 3DPVT
Kars
ten
Mül
ler
Slide 53
18.03.2010 Universität Klagenfurt
3D Video Coding Group in MPEG
• Nagoya Univ. • NTT• UPM• Philips• GIST• ETRI• JVC• Poznan Univ. of
Technology• Thomson• MERL• Peking Universtity• Tsinghua Univ.• Huawei• Xidian University• Telefonica
• Samsung• LG Electronics• Nokia• NICT• Kwangwoon Univ.• Sharp• Fraunhofer• NCTU/ITRI• Motorola• NXP• Ericsson• Logitech• Orange/France
Telecom • FUB• Sony
• Zhejiang Univ.• ASTRI• Qualcomm
• Coordination:• Karsten Müller (HHI)
Anthony Vetro (MERL)
Kars
ten
Mül
ler
Slide 54
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
3D@Home Consortium
• Steering Team 1: Content Creation & ProductionObjective: To focus on projects which improve the quality of 3D content for the home audience.
• Steering Team 2: Content Storage, Transmission & DistributionObjective: To develop useful definitions and guidelines for the successful storage, transmission and distribution of 3D content.
• Steering Team 3: 3D PromotionObjective: To promote the adoption of 3D into the home by promoting 3D format and the 3D@Home Consortium.
• Steering Team 4: 3D DisplaysObjective: To identify issues around 3D Displays and associated hardware.
Slide 55
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
3D@Home Consortium
Slide 56
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
EU 3D, Immersive, Interactive Media Cluster• Currently, a number of EU projects develop parts and products of the 3D
media chain for certain application areas, e.g. 3D home entertainment or mobile services
• Enhanced success of these EC funded projects and subsequent associated commercialization efforts if simultaneous operations of individual projects can be further orchestrated towards a strong mutual presence
• Better highlight of Europe‘s good position in worldwide R&D
• 3D, Immersive, Interactive Media Cluster is an umbrella structure for the projects and provides international contact and information about project activities
• 3D Media, Immersive, Interactive Cluster is one of the main drivers towards "3D Media" in the context of the "Future Internet" activities supported by EU Commission (DG INFSO)
Slide 57
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
3DII Media Cluster – Current Projects
• Leadership: Prof. Levent Onural and Dr. Karsten Müller
• Projects: Status: Framework Program• 3DTV ended 2008 FP6• 3DPHONE ongoing FP7• MOBILE3DTV ongoing FP7• 3D4YOU ongoing FP7• 2020 3D Media ongoing FP7• 3DPresence ongoing FP7• MUTED ended 2008 FP6• HELIUM3D ongoing FP7• Real3D ongoing FP7• i3DPost ongoing FP7• VICTORY ended 2009 FP7
3D for MobileSystems
3D for BroadcastSystems
3D DisplayTechnology
3D ContentDescription
Slide 58
3D Videoconference
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
3DII Media Cluster – New Projects
• Projects: Status: Framework Program• DIOMEDES new 2010 FP7• MUSCADE new 2010 FP7• 3D VIVANT new 2010 FP7• BEAMING new 2010 FP7• FINE new 2010 FP7• SKYMEDIA new 2010 FP7• SALA3D new 2010 FP7• 3DLife new 2010 FP7• FascinatE new 2010 FP7
Slide 59
18.03.2010 Universität Klagenfurt
Kars
ten
Mül
ler
Summary
• 3D technology is maturing due to world wide development from capturing to display
• First systems are in use, mainly using stereoscopic displays and MVC coding
• New challenging research topics in all areas of 3D video:– Reliable capturing devices with synchronized views– Improved time-consistent depth estimation– Geometry-enhanced multi-view coding (e.g. 3DVC)– Robust view synthesis– New high-resolution multi-view displays
(e.g. 50 views, each with HDTV resolution!)
Slide 60
18.03.2010 Universität Klagenfurt