Three-Dimensional VideoPostproduction and Processing

Ibraheem Alhashim - July 10th 2013CMPT 880

2

Overview

History + Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook

3

Overview

History + Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook

4

A bit of history

Imaging technology

James Maxwell 1855

JC d'Almeida 1858Joseph Niépce 1826 

Stereoscope 1860

Underwood 1901

The Power of Love (1922 film)

The Jazz Singer 1927

5

A bit of history

Becky Sharp (1935) 1950s

6

A bit of history

Timeline of 3D movies

1950s 1980s 2010s

Worldwide: $2,782,275,172

7

3D Feature Films Highest-grossing films 2012: nine of the top 15 were

in 3D [businessinsider.com]

Industry forecast > 20% of TVs by 2015

Year 3D Films2005 42006 72008 82009 202010 302011 382012 38

8

Other uses

WW2 maps Virtual reality 3D Electron Microscopy Video games + virtual cinema 1968

1945

198020132013 2013

9

3D Video

Usually marketed as objects popping off screen

10

3D Video

In reality.. It’s the same old concept Present slightly different image per eye The brain combines them and perceives depth

Trick human visual system Stereo 3D content production:

technical, psychological, and creative skills

11

3D Video - Issues

Not as straightforward as 2D production Several considerations for a good 3D experience Balance between 3D effect and overall experience Minimize viewing discomfort

Stereoscopic comfort zone Scene depth adaptation Control of global and local disparity Video composition

12

Stereoscopic Comfort Zone Comfort zone

Stereographer “bring the whole real world inside this virtual space”

13

Control of Absolute Disparity Convergence is controlled by shifting the views

14

Scene Depth Adaptation

Keep in mind disparity range (screen size and resolution)

Carefully plan a scene’s 3D effect Consider transitions and provide resting periods Post-production depth adaptation

Manual changes per display

15

3D Display Adaptation From cinema to TV Depth composition has to modified (stereographer)

Depth information allows for virtual view interpolation

16

Local Disparity Adaptation Objects should be within stereoscopic window

Intentional depth changes Physical / multiple camera rigs Synthetic / depth-editing

Objects at the border Can cause retinal rivalry Should be cropped by virtually shifting screen plane closer However, not applicable to live broadcast

17

Live auto-correction

Automatic correction & manipulation of stereo live broadcast Live sport events (big player) Close objects could abruptly appear

Open research problem Some kind of novel view synthesis

18

Video composition

Mixing and Composition of 3D Material, Real and Animated Content Graphics overlay / subtitles Cannot be simply pasted over other footage Leverage knowledge about depth range of footage

19

3D Video - Issues

Summary 3D Production has an “art” component Different medium requires different parameters Content makers / directors need to think about 3D issues Stereographer try to balance 3D effect with overall

viewing experience

20

3D video

Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook

21

3D Display Technologies

“Offer immersive experience” 3D Glasses (cinema + TV) Head-mounted displays Volumetric and holographic displays Autostereoscopic displays

22

Autostereoscopic displays

Best choice for mobile devices Backward compatible & closer to viewer expectation Most common

Parallax barrier Lenticular sheet

23

Autostereoscopic displays Crosstalk (ghosting) is most important parameter

“information meant for one eye intrudes into the other eye’s view”

24

Autostereoscopic displays Issues

Generally less available depth range More ghosting artifacts Also, depth information is essential

Synthesize different views

25

3D video

Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook

26

Basic processing Signal processing to avoid visual artifacts Any small visual discrepancy could cause discomfort

Three main categories1. Correction of Geometrical Distortions2. Color Matching3. Adjustment of Stereo Geometry

27

Correction of Geometrical Distortions

Camera rigs might not be perfectly aligned Real lenses impose radial distortions by nature Other lens parameters might not sync E.g. geometrical lens distortions or chromatic

aberration

28

Color Matching Color discrepancy can lead to eyestrain and visual

fatigue Manual calibration need to be done on cameras Automatic methods exist (histogram filtering)

29

Color Matching

Modern professional postproduction tools incorporate stereo color matching and grading

30

Adjustment of Stereo Geometry

Convergence need to be selected and balanced carefully to achieve good stereo content

Usually by shifting images horizontally in contrary directions, however, cropping & scaling is needed

“shift-crop-scale” Demo http://stereo.nypl.org/create

31

Adjustment of Stereo Geometry

Stereo baseline is fixed during shooting Several hardware solutions help camera team

analyze the disparity range Also help visualize result of shift-crop-scale

32

3D video

Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook

33

3D Depth information

Depth information is needed for novel output images in post-production adjusting the view parallax (different screens) many different uses in postproduction

34

Depth info

Extracting depth information (time of flight camera / SfS)

Structure-from Stereo (SfS) Advanced computer vision problem Stereo matching

Local – block matching, optical flow est. Global – graph cuts, simulated annealing

35

Structure-from Stereo

Global methods are more accurate Slow + don’t work well on video / motion

Local methods are more widely used Window-based methods Some system are in real-time Blocky output

36

Example of Depth-based method

Apply hybrid recursive matching (HRM) Follow by cross-trilateral median filtering (ACTMF) Semi-automatic

37

Post processing depth

Align depth discontinues to object borders Remove noise and mismatches Fill occlusions Approaches

Use image segmentation Neighborhood filtering

38

View synthesis

Synthesize new virtual stereo views by image-based rendering

Input – depth + color Output – image with new view

Two types depth-based + warping-based

39

Depth-based

Computer vision techniques Image-based rendering (IBR) Depth-based rendering (DIBR) Layered-depth images (LDI) Intermediate view reconstruction (IVR)

40

Depth-Image-Based Rendering

Need pixel-by-pixel depth maps Recent focus

Handle depth discontinues Better depth boundaries

41

Warping-based Methods that deform the image content directly Compress or stretch by nonlinear warp function Do not need camera calibration, segmentation, fill holes Worst case, visible wobbling artifact

42

View synthesis

Summary Depth maps are computed using computer vision

techniques (still active) Generate new views by image-based rendering or warping Warping methods can potentially have less visual problems

DEMO

43

3D video

Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook

44

2D to 3D

3D to 2D is trivial Hot topic for 3DTV and 3D cinema Methods so far are

Manual (computer assisted) Automatic

45

Depth cues

Human visual systemMonocular cuesBinocular cues

46

Depth cues

Monocular depth cues - things can be seen by one eye (2D Camera)

Focus / defocus, perspective, relative size Light and shading and texture Motion parallax

Binocular – 2 eyes or 3D camera rigs accommodation, convergence, and binocular discrepancy

47

Manual 2D to 3D

Applicable to prerecorded video Utilize depth cues to generate a stereoscopic view

for each frame Time consuming and costly Cost vs. quality

48

Manual 2D to 3D Three major steps

Rotoscoping / segmentation Depth assignment Inpainting

Few companies provide process as a service

49

Depth Assignment

Shifting different parts of scene to simulate 3D To avoid cardboard effect hire a “3D compositor”

Create displacement maps for each pixel Use 3D primitives, spheres or cubes Use DIBR or 3D warping to synthesize view cannot handle transparencies well

50

Occlusion filling

Also known as “in-painting” One of the most challenging parts in 2D/3D

conversion

51

Automatic conversion

Automatic systems Extract depth information + synthesize stereoscopic images

52

Automatic segmentation

Automatic segmentation of background Background subtraction techniques (ML) Optical-flow-based (uses motion as cue)

Objects in scene segmentation Contour-based (color, edges, texture) “SnapCut” state of the art algorithm in After Effects

53

Automatic depth estimation 3D Structure recovery (Shape From X)

Depth from focus/defocus Depth from geometric cues

Relative sizes, gradients Depth from color and intensity

Lights and shadow Depth from motion

Motion parallax

54

Automatic in-painting

Relay on - texture synthesis or border continuation Some work is done on spatio-temporal inpainting

Enforce global temporal consistency for patches Object motion could reveal background

Not viable in practice

55

Real-time and Offline conversion

Real-time Approximate using motion parallax Could use color + intensity information Height-depth cues Hybrid approaches are better

Offline Apply structure from motion Create virtual scene using different views Slow cloudification?

56

2D to 3D

Summary Hire specialist to rebuild depth and frame objects Relay on computer vision methods to build depth info Fill background using in-painting Still an active research topic

segmentation + depth estimation + synthesis

57

3D video

Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook

58

3D media for mobile devices

Consider size of display Computing power + Network constraints Encoding

3D Video coding (V+D) Multiview Video Coding (MVC)

59

Issues

Stereoscopic images have different disparity Coding artifacts

Dealing with different channels Transmission artifacts

Image distortion on loss Temporal mismatch

Display artifacts autostereoscopic display problems (ghosting)

60

3D media for mobile devices Not only mobile devices, Internet connected devices Standards for 3D video encoding are still developing

Challenges for 2D video on mobile and more (depth)

3D Video conferencing demand real-time processing

Display technologies are evolving Resolution, need an adaptive and future ready format

61

3D video

Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook

62

Outlook

Today, 3D video technology are becoming practical Content creation process have matured

still a lot of room for improvement and extension There is a huge commercial incentive in this field A lot of areas to work on…

63

Outlook

64

Commentary

3D has been historically about generating revenue using mind tricks

Tricks are getting much better! Immersive experience ≠ quality

65

Commentary

Future of 3D Video Consumers decide it Good for short media If all devices support it

Cinema products areprofitable!

66

Commentary Similar strategies could apply on different technology

Light-field camera – Lytro first product 2011 (2006) Nokia / Pelican Imaging 16-lens array camera 2014

67 Thank you!

68

Video demo

http://www.youtube.com/watch?v=pEXSkRpL0dI