The Virtual Photo Set (VPS) · Project results Scientiﬁc output-13 journal papers -39...

The Virtual Photo Set (VPS) IIS11-0081: Data-driven scene characterization

for realistic rendering

2017-11-14

Project overview

Project title: IIS11-0081 Data-driven scene characterisation for realistic rendering

Project leader: Anders Ynnerman and Jonas Unger

Principal investigators: Michael Felsberg, Fredrik Gustafsson, Reiner Lenz, Jonas Unger, Anders Ynnerman

Funding amount: 27,000,000 SEK Funding period: 2012-01-01 - 2016-12-31

The image paradigm shift

The vision of this project is to develop the foundation for the next generation imaging pipelines where reality can be edited and virtual and real objects can be seamlessly mixed.

Application drivers

• Digital design

• Product visualization

• Special effects in movies

• Computer games

• Augmented reality

• ...

Major challengesA High Dynamic Range (HDR) imaging pipeline

• Multi-modal input data • Tracking of input devices • TBs of data per capture • On-line user feedback

Processing and editing RenderingScene Capture

• Geometry and light source extraction, and scene editing

• Data representations and compression

• Interactive processing and user feedback

• Ultra-realistic off-line rendering

• Efficient material models • Compression for real-time

applications • Tone mapping

Project overviewProject team:C-Research, ITN, LiU, (Prof. Anders Ynnerman) Computer Vision Laboratory, ISY, LiU, (Prof. Michael Felsberg) Sensor Fusion Group, ISY, LiU, (Prof. Fredrik Gustafsson) 6 senior researchers, 6 PhD students, and 4 research engineers

Collaborations

• Linköping University • University of Southern California • Warwick University • Bangor University

• IKEA Communications • SpheronVR AG • IrysTech • Swiss International • Volvo PVH

Scientific collaborations Industry collaborations

New projects, funding proposals, and joint papers with new both academic and industrial partners

Project resultsScientific output

- 13 journal papers - 39 contributions at leading conferences in the field - 3 book chapters - 2 PhD theses (supported by this project) + 2 theses spring 2018 - 1 Licentiate thesis - 5 M.Sc. theses

Open data and software- HDR-video sequence data sets for research and educational use - Lighting environments captured using HDR imaging - Depends workflow management system - LumaHDRv high dynamic range (HDR) video codec

URLs: [ www.hdrv.org ], [ www.dependsworkflow.net ], [ www.lumahdrv.org ]

Spin-offs - Materialeyes AB (measurement systems, methods, and

algorithms for appearance capture ) - MassVis AB

http://www.hdrv.org

http://www.dependsworkflow.net

http://www.lumahdrv.org

VPS solutions driving the state-of-the-art

Image based lighting

Current state-of-the-art: Lighting is captured as a still image at a single position and at a single instant in time

HDR Light Probes

Virtual scene

Virtual

camera

HDR environment

map

L(x, ~!o

) =

Z

⌦Lenv

(~!i

)⇢(x, ~!i

! ~!o

)(�~!i

· ~nx

)V (~!i

)d~!i

- Lighting is described in the panoramic HDR image- Each pixel corresponds to the scene radiance incident over the solid angle subtended by the pixel

Lighting Material Cosine falloff Visibility

Mixing Virtual and Real

L(x, ~!o

) =

Z

⌦Lenv

(~!i

)⇢(x, ~!i

! ~!o

)(�~!i

· ~nx

)V (~!i

)d~!i

Lighting Material Cosine falloff Visibility

Requires high dynamic range (HDR) videoand light re-projection onto scene geometry

Rendering using accurately measured lighting recovered using the VPS approach

Image synthesis using previous state-of-the-art methods

Challenges (and VPS solutions)Capture: lighting conditions, scene geometry, and material information

Light Fields

Capture: lighting conditions, scene geometry, and material information

- Robust capture of lighting conditions in complex environments

- High resolution HDR-video beyond full HD

- Effective dynamic range at least 1:10,000,000

- Accurate radiometric calibration

LiU HDRv: http://www.hdrv.org

Our demands on HDR-video go far beyond commercial solutions

Need for HDR Video

Challenges

http://www.hdrv.org

Affine Transform

Affine Transform

f(x,y)PSF

Sampling(R,G,B) +

d(x,y)p(x,y)

Affine Transform Sampling(R,G,B) +

ND Filter

ND Filter

Sampling(R,G,B) +ND Filter

F

T1

T2

T3

I2

I3

I1

Zj


- Large scale data: ~1.5GB/s of floating point pixel data

- Minutes of capture leads to TBs of data - Real-time user feedback and HDR image

reconstruction is a requirement - VPS result: Novel image reconstruction

framework for multi-sensor systems suitable for parallel computations and GPU implementation

- Active in COST Action IC1005: Capture, storage, transmission and display of real-world lighting

HDR capture device

Joel Kronander, Stefan Gustavson, Gerhard Bonnet, Anders Ynnerman, Jonas Unger, "A unified framework for multi-sensor HDR video reconstruction", Signal Processing : Image Communications, 29(2): 203-215, 2014.

J. Kronander, S. Gustavson, G. Bonnet, A. Ynnerman, J. Unger: Unified HDR Reconstruction from RAW CFA Data, In proceedings of the IEEE International Conference on Computational Photography (ICCP), 2013

Light Reprojection

• Project HDR light fields onto 3D geometry • Efficient representation for data compression • Encodes high frequency features • User level editing of geometry, materials and light sources



Our demands on the model poses research challenges in tracking and geometry estimation

Scene reconstruction

- Millimeter accuracy in recovered model - Submillimeter accuracy and sub degree

accuracy in camera pose and trajectory estimation

- Fusion of image information and range data - Robustness to non-stationary objects in the

scene

Challenges

Light source extraction

J. Unger, J. Kronander, P. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR-video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.

J. Unger, J. Kronander, P. Larsson, S. Gustavson, A. Ynnerman, Temporally and Spatially Varying Image Based Lighting using HDR-video, Proceedings of EUSIPCO '13, 2013.

Image synthesis



Captured Scene

Tables removed

Project HDR data

Panoramic image displaying layout of the scene

Recovered model



Room populated with virtual furnitureJ. Unger, J. Kronander, P. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR-video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.


Norrköping

Computer graphics imageJ. Unger, J. Kronander, P. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR-video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.


Norrköping

J. Unger, S. Gustavson, J. Kronander, P. Larsson, G. Bonnet, and G. Kaiser. 2011. Next generation image based lighting using HDR video. In ACM SIGGRAPH 2011 Talks


Computer graphics image

Norrköping

Computer graphics imageJ. Unger, J. Kronander, P. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR-video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.


Surface light fields

Store the light field at surfaces in the scene

Learning based compression using Exemplar Orthogonal Bases

Ehsan Miandji, Joel Kronander, Jonas Unger, "Learning Based Compression of Surface Light Fields for Real-time Rendering of Global Illumination Scenes", Proceedings of ACM SIGGRAPH ASIA 2013, 2013.

Real-time rendering

Compression

Compressed rendering

Ehsan Miandji, Joel Kronander, Jonas Unger, “Image reconstruction in reduced union of subspaces”, submitted to Eurographcis 2015

Light field reconstruction

Ehsan Miandji, Joel Kronander, Jonas Unger. Compressive image reconstruction in reduced union of sub-spaces. Computer Graphics Forum special issue Proceedings of Eurographics '15, Vol. 34, No. 2, 2015

Demonstrator integrationDepends workflow

management system• Generality

• Fusion of data from different input modalities• Modularity

• The ability to reuse software components and customize the processing to specific problems

• Data provenance • A highly important aspect is to enable the ability to go back

and easily determine what generated certain results.• Performance

• Large scale input data sizes (often in the order of TBs) and the computational complexity of the processing algorithms present significant challenges.

• Interaction• Advanced visualization and artistic interaction to steer the

computations.

Andrew Gardner, Jonas Unger, "Depends: Workflow Management Software for Visual Effects Production", Digital Production Symposium, DigiPro '14, Vancouver, Canada, August, 2014, DigiPro '14, 2014.

A Swedish House

LibraryC++ API for encoding and decoding of HDR video ApplicationsEncoding of HDR video frames Decoding of HDR video Playback of HDR video DependenciesMatroska media container VP9 (libvpx) from Google

Gabriel Eilertsen, Jonas Unger, and Rafal K. Mantiuk. Luma HDRv: an open source high dynamic range video codec optimized by large-scale testing. Accepted for SIGGRAPH '16 Talks, Annaheim, USA, 2016

Gabriel Eilertsen, Rafal K. Mantiuk, Jonas Unger. A High Dynamic Range Video Codec Optimized by Large Scale Testing, IEEE International Conference on Image Processing '16, Phoenix, USA, September, 2016.

HDR video compression



Real-time noise aware TMODisplay: adapt input video to display characteristics, viewing environment and perception

HDREdge-stopping

spatial filter (fast detail

extraction diffusion) Local tone curves (noise aware minimum

contrast distortion)

Input HDR-video

Parameters: Peak luminance, Dynamic range, and Ambient light measurements

Noise-awarecontrol over

image details

Parameters: Local/global, Tone compression, Exposure, and Noise visibility control

Parameters: Edge stop

Parameter: Detail scaling, Noise visibility control

Tone-mapped output

Data flowUser parameterDisplay parameterNoise estimate

Noise model

Detail layer,

Base layer,

Input frame, Inverse

display model

Real-time noise aware tone mapping operator

Gabriel Eilertsen, Rafal Mantiuk, and Jonas Unger, Real-time noise-aware tone mapping, Accepted for publication in ACM Transactions on Graphics proceedings of Siggraph Asia '15, Kobe, Japan November, 2015

Glasses free 3D display

Andrew Jones, Jonas Unger, Koki Nagano, Jay Busch, Xueming Yu, Hsuan- Yueh Peng, Oleg Alexander, Mark Bolas, Mark and Paul Debevec. An Automultiscopic Projector Array for Interactive Digital Humans. In ACM SIGGRAPH 2015 Emerging Technologies, August 2015.

Andrew Jones, Jonas Unger, Jay Busch, Xueming Yu, Hsuan-Yueh Peng, Oleg Alexander and Paul Debevec. Creating a life-sized automultiscopic Morgan Spurlock for CNN's ``Inside Man''. In ACM SIGGRAPH 2014 Talks, Vancouver, Canada, August, 2014.


335.28 CM

121.92 CM

.63 DEGREES

216 PROJECTORS

SCREEN

385 CM

6 DEGREES

CAMERAS

216 projectors

0.63 degrees

Projection screen 250 x 140 cm

340 cm Projectors

The Virtual Photo Set (VPS) · Project results Scientiﬁc output-13 journal papers -39...

Documents

Transcript of The Virtual Photo Set (VPS) · Project results Scientiﬁc output-13 journal papers -39...